[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:37:31 dist-test-slave-rxv6 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:37:31 dist-test-slave-rxv6 krb5kdc[399](info): set up 2 sockets
Jun 19 18:37:31 dist-test-slave-rxv6 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 18:37:32.338646 392 mini_kdc.cc:121] Time spent starting KDC: real 2.435s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260619 18:37:32.543195 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.204s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260619 18:37:32.702975 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.160s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260619 18:37:32.827466 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.124s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/test-user.keytab.
W20260619 18:37:33.468859 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.641s user 0.000s sys 0.002s
Jun 19 18:37:33 dist-test-slave-rxv6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:37:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:37:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260619 18:37:33.945582 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.450s user 0.000s sys 0.004s
I20260619 18:37:33.946861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42395
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:34.078253 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:34.078616 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:34.078662 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:34.082855 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:37:34.082944 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:34.082967 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:34.082986 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:37:34.083003 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:37:34.088831 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:34.090155 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:34.091760 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:34.104147 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:34.104185 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:34.104187 412 server_base.cc:1034] running on GCE node
W20260619 18:37:34.104398 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:34.105096 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:34.106482 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:34.107635 412 hybrid_clock.cc:648] HybridClock initialized: now 1781894254107584 us; error 71 us; skew 500 ppm
Jun 19 18:37:34 dist-test-slave-rxv6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:34.110842 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:37:34.111882 412 webserver.cc:460] Webserver started at http://127.0.98.62:42187/ using document root <none> and password file <none>
I20260619 18:37:34.112614 412 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:34.112684 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:34.112928 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:34.115476 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "7a1c7d13fef7415cb0dd81da0c03b0a8"
format_stamp: "Formatted at 2026-06-19 18:37:34 on dist-test-slave-rxv6"
server_key: "2c43e02afa7e0b62a557f4ebbce340ac"
server_key_iv: "321d3b528adb06ebc4812fb86cf5ddca"
server_key_version: "encryptionkey@0"
I20260619 18:37:34.116107 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "7a1c7d13fef7415cb0dd81da0c03b0a8"
format_stamp: "Formatted at 2026-06-19 18:37:34 on dist-test-slave-rxv6"
server_key: "2c43e02afa7e0b62a557f4ebbce340ac"
server_key_iv: "321d3b528adb06ebc4812fb86cf5ddca"
server_key_version: "encryptionkey@0"
I20260619 18:37:34.120496 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:37:34.123308 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:34.124815 412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:37:34.124948 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "7a1c7d13fef7415cb0dd81da0c03b0a8"
format_stamp: "Formatted at 2026-06-19 18:37:34 on dist-test-slave-rxv6"
server_key: "2c43e02afa7e0b62a557f4ebbce340ac"
server_key_iv: "321d3b528adb06ebc4812fb86cf5ddca"
server_key_version: "encryptionkey@0"
I20260619 18:37:34.125068 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:34.154192 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:34.157817 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:34.158095 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:34.166446 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42395
I20260619 18:37:34.166479 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42395 every 8 connection(s)
I20260619 18:37:34.167644 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:37:34.171136 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:34.175819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 412
I20260619 18:37:34.176024 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:37:34.177878 392 external_mini_cluster.cc:1428] Setting key 0669ca00d05421488f7ddec196c96a86
I20260619 18:37:34.178606 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8: Bootstrap starting.
I20260619 18:37:34.182090 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:34.183022 480 log.cc:824] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:34.185175 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8: No bootstrap required, opened a new log
I20260619 18:37:34.189844 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1c7d13fef7415cb0dd81da0c03b0a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42395 } }
I20260619 18:37:34.190214 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:34.190287 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a1c7d13fef7415cb0dd81da0c03b0a8, State: Initialized, Role: FOLLOWER
I20260619 18:37:34.190852 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1c7d13fef7415cb0dd81da0c03b0a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42395 } }
I20260619 18:37:34.191002 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:37:34.191068 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:37:34.191172 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:34.192303 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1c7d13fef7415cb0dd81da0c03b0a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42395 } }
I20260619 18:37:34.192679 480 leader_election.cc:302] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a1c7d13fef7415cb0dd81da0c03b0a8; no voters:
I20260619 18:37:34.193014 480 leader_election.cc:288] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:37:34.193113 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:34.193326 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7a1c7d13fef7415cb0dd81da0c03b0a8, State: Running, Role: LEADER
I20260619 18:37:34.193677 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1c7d13fef7415cb0dd81da0c03b0a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42395 } }
I20260619 18:37:34.194526 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:37:34.195410 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a1c7d13fef7415cb0dd81da0c03b0a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1c7d13fef7415cb0dd81da0c03b0a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42395 } } }
I20260619 18:37:34.195567 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:34.195919 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a1c7d13fef7415cb0dd81da0c03b0a8. Latest consensus state: current_term: 1 leader_uuid: "7a1c7d13fef7415cb0dd81da0c03b0a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a1c7d13fef7415cb0dd81da0c03b0a8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42395 } } }
I20260619 18:37:34.196012 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:34.196290 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:37:34.199935 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 18:37:34 dist-test-slave-rxv6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:34.206770 494 catalog_manager.cc:1347] Generated new cluster ID: 64821389419f499782bfd9e0f5093f29
I20260619 18:37:34.206835 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:37:34.210450 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:34.182069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.0.98.62:42395)
0619 18:37:34.182691 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:34.182716 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:34.184052 (+ 1336us) server_negotiation.cc:407] Connection header received
0619 18:37:34.189201 (+ 5149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:34.189232 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:34.189748 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:34.190240 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:34.192992 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:34.194511 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:34.196402 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:34.197149 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:34.197309 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:34.202159 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:34.202164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:34.202178 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:34.202207 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:34.206719 (+ 4512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:34.207416 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:34.207422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:34.207426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:34.207542 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:34.207907 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:34.207911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:34.207913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:34.208347 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:34.208546 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:34.209141 (+ 595us) server_negotiation.cc:299] Negotiation successful
0619 18:37:34.209418 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":200,"threads_started":1}
I20260619 18:37:34.223901 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:37:34.224884 494 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:37:34.230247 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a1c7d13fef7415cb0dd81da0c03b0a8: Generated new TSK 0
I20260619 18:37:34.230997 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:37:34.265730 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42395
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:34.426761 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:34.427125 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:34.427173 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:34.431527 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:34.431620 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:34.431699 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:37:34.437500 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:34.438970 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:34.440292 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:34.451329 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:34.451367 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:34.451490 506 server_base.cc:1034] running on GCE node
W20260619 18:37:34.451722 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:34.452172 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:34.452943 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:34.454108 506 hybrid_clock.cc:648] HybridClock initialized: now 1781894254454081 us; error 42 us; skew 500 ppm
Jun 19 18:37:34 dist-test-slave-rxv6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:34.457151 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:37:34.458247 506 webserver.cc:460] Webserver started at http://127.0.98.1:44963/ using document root <none> and password file <none>
I20260619 18:37:34.458897 506 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:34.458967 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:34.459211 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:34.461642 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "a041c2461df84b93b3e081be7d056160"
format_stamp: "Formatted at 2026-06-19 18:37:34 on dist-test-slave-rxv6"
server_key: "bd675faea90ce1f3229cbb3438a47259"
server_key_iv: "abd35a0b13795d4fbdb9755a1be444c4"
server_key_version: "encryptionkey@0"
I20260619 18:37:34.462297 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a041c2461df84b93b3e081be7d056160"
format_stamp: "Formatted at 2026-06-19 18:37:34 on dist-test-slave-rxv6"
server_key: "bd675faea90ce1f3229cbb3438a47259"
server_key_iv: "abd35a0b13795d4fbdb9755a1be444c4"
server_key_version: "encryptionkey@0"
I20260619 18:37:34.466753 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:37:34.469695 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:34.471240 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:37:34.471369 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "a041c2461df84b93b3e081be7d056160"
format_stamp: "Formatted at 2026-06-19 18:37:34 on dist-test-slave-rxv6"
server_key: "bd675faea90ce1f3229cbb3438a47259"
server_key_iv: "abd35a0b13795d4fbdb9755a1be444c4"
server_key_version: "encryptionkey@0"
I20260619 18:37:34.471487 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:34.491421 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:34.495534 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:34.495770 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:34.496554 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:34.497591 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:34.497644 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:34.497705 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:34.497741 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:34.508605 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45627
I20260619 18:37:34.508628 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45627 every 8 connection(s)
I20260619 18:37:34.509739 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:37:34.514513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 506
I20260619 18:37:34.514621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:37:34.514889 392 external_mini_cluster.cc:1428] Setting key 974d75848326cbd908b6911e128e5873
Jun 19 18:37:34 dist-test-slave-rxv6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 18:37:34.526533 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:34.512236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56741 (local address 127.0.98.62:42395)
0619 18:37:34.512436 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:34.512441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:34.513402 (+ 961us) server_negotiation.cc:407] Connection header received
0619 18:37:34.515028 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:34.515064 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:34.515157 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:34.515258 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:34.516762 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:34.517539 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:34.518820 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:34.519178 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:34.519282 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:34.522367 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:34.522372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:34.522375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:34.522409 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:34.524096 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:34.524678 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:34.524683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:34.524685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:34.524783 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:34.525172 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:34.525175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:34.525177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:34.525436 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:34.525552 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:34.526219 (+ 667us) server_negotiation.cc:299] Negotiation successful
0619 18:37:34.526357 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 18:37:34.527662 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:34.512652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42395 (local address 127.0.98.1:56741)
0619 18:37:34.513222 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:34.513280 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:34.514767 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:34.515463 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:34.515473 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:34.515932 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:34.516606 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:34.516625 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:34.517653 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:34.517655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:34.518701 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:34.518712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:34.519337 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:34.519340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:34.519508 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:34.520237 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:34.520256 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:34.522164 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:34.524233 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:34.524238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:34.524257 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:34.524554 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:34.524885 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:34.524887 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:34.524910 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:34.525071 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:34.525593 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:34.525599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:34.525936 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 18:37:34.526150 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 18:37:34.526524 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 18:37:34.529234 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42395
I20260619 18:37:34.529553 635 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:34.530296 635 heartbeater.cc:511] Master 127.0.98.62:42395 requested a full tablet report, sending...
I20260619 18:37:34.532286 444 ts_manager.cc:194] Registered new tserver with Master: a041c2461df84b93b3e081be7d056160 (127.0.98.1:45627)
I20260619 18:37:34.533664 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:37:34.545217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42395
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:34.673488 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:34.673815 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:34.673863 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:34.678150 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:34.678222 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:34.678292 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:37:34.683974 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:34.685353 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:34.686668 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:34.698225 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:34.698225 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:34.698225 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:34.698524 640 server_base.cc:1034] running on GCE node
I20260619 18:37:34.698983 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:34.699800 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:34.700973 640 hybrid_clock.cc:648] HybridClock initialized: now 1781894254700943 us; error 43 us; skew 500 ppm
Jun 19 18:37:34 dist-test-slave-rxv6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:34.703967 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:37:34.704999 640 webserver.cc:460] Webserver started at http://127.0.98.2:39905/ using document root <none> and password file <none>
I20260619 18:37:34.705652 640 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:34.705711 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:34.705940 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:34.708443 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "a03af66c7daa4f68aac807435174e178"
format_stamp: "Formatted at 2026-06-19 18:37:34 on dist-test-slave-rxv6"
server_key: "ef0dfaddf507c863339a4842a3b925d4"
server_key_iv: "9b8f1b4d5ae0627a1376f84e7f601692"
server_key_version: "encryptionkey@0"
I20260619 18:37:34.709059 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a03af66c7daa4f68aac807435174e178"
format_stamp: "Formatted at 2026-06-19 18:37:34 on dist-test-slave-rxv6"
server_key: "ef0dfaddf507c863339a4842a3b925d4"
server_key_iv: "9b8f1b4d5ae0627a1376f84e7f601692"
server_key_version: "encryptionkey@0"
I20260619 18:37:34.713514 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:37:34.716490 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:34.718042 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:37:34.718165 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "a03af66c7daa4f68aac807435174e178"
format_stamp: "Formatted at 2026-06-19 18:37:34 on dist-test-slave-rxv6"
server_key: "ef0dfaddf507c863339a4842a3b925d4"
server_key_iv: "9b8f1b4d5ae0627a1376f84e7f601692"
server_key_version: "encryptionkey@0"
I20260619 18:37:34.718286 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:34.737493 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:34.741196 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:34.741395 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:34.742153 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:34.743300 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:34.743355 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:34.743404 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:34.743433 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:34.754626 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36583
I20260619 18:37:34.754647 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36583 every 8 connection(s)
I20260619 18:37:34.755716 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:37:34.763253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 640
I20260619 18:37:34.763399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:37:34.763715 392 external_mini_cluster.cc:1428] Setting key c527d0f7df2de24919b0626889930ffe
Jun 19 18:37:34 dist-test-slave-rxv6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:34.772230 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:34.757983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57075 (local address 127.0.98.62:42395)
0619 18:37:34.758143 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:34.758147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:34.759214 (+ 1067us) server_negotiation.cc:407] Connection header received
0619 18:37:34.760836 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:34.760839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:34.760900 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:34.760980 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:34.762480 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:34.763082 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:34.764737 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:34.765088 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:34.765212 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:34.768237 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:34.768241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:34.768243 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:34.768269 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:34.769710 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:34.770288 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:34.770291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:34.770292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:34.770371 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:34.770712 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:34.770715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:34.770716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:34.770912 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:34.770998 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:34.771571 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 18:37:34.772011 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":47}
I20260619 18:37:34.773703 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:34.758476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42395 (local address 127.0.98.2:57075)
0619 18:37:34.759019 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:34.759071 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:34.760516 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:34.761176 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:34.761186 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:34.761621 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:34.762305 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:34.762322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:34.763217 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:34.763220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:34.764588 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:34.764600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:34.765251 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:34.765255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:34.765464 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:34.766203 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:34.766225 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:34.768054 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:34.769823 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:34.769829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:34.769850 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:34.770185 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:34.770451 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:34.770454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:34.770456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:34.770619 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:34.771009 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:34.771016 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:34.771317 (+ 301us) client_negotiation.cc:769] Sending connection context
0619 18:37:34.771503 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 18:37:34.772281 (+ 778us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":122,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 18:37:34.775359 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42395
I20260619 18:37:34.775647 769 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:34.776340 769 heartbeater.cc:511] Master 127.0.98.62:42395 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 18:37:34.777698 444 ts_manager.cc:194] Registered new tserver with Master: a03af66c7daa4f68aac807435174e178 (127.0.98.2:36583)
I20260619 18:37:34.778355 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:37:34.793630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42395
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:34.911959 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:34.912231 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:34.912276 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:34.916255 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:34.916316 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:34.916374 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:37:34.921698 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42395
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:34.922855 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:34.923954 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:34.934108 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:34.934079 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:34.934137 774 server_base.cc:1034] running on GCE node
W20260619 18:37:34.934085 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:34.934718 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:34.935371 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:34.936538 774 hybrid_clock.cc:648] HybridClock initialized: now 1781894254936515 us; error 39 us; skew 500 ppm
Jun 19 18:37:34 dist-test-slave-rxv6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:34.943259 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:37:34.944183 774 webserver.cc:460] Webserver started at http://127.0.98.3:35927/ using document root <none> and password file <none>
I20260619 18:37:34.944763 774 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:34.944828 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:34.945020 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:34.947115 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "ab2fe314585743f0a6f3ada1580d4d47"
format_stamp: "Formatted at 2026-06-19 18:37:34 on dist-test-slave-rxv6"
server_key: "434bfdfe975f93cfea876b8ebe319002"
server_key_iv: "7172525bc091e21fb424aabd67c0e858"
server_key_version: "encryptionkey@0"
I20260619 18:37:34.947652 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ab2fe314585743f0a6f3ada1580d4d47"
format_stamp: "Formatted at 2026-06-19 18:37:34 on dist-test-slave-rxv6"
server_key: "434bfdfe975f93cfea876b8ebe319002"
server_key_iv: "7172525bc091e21fb424aabd67c0e858"
server_key_version: "encryptionkey@0"
I20260619 18:37:34.951531 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:37:34.954185 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:34.955566 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:37:34.955688 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "ab2fe314585743f0a6f3ada1580d4d47"
format_stamp: "Formatted at 2026-06-19 18:37:34 on dist-test-slave-rxv6"
server_key: "434bfdfe975f93cfea876b8ebe319002"
server_key_iv: "7172525bc091e21fb424aabd67c0e858"
server_key_version: "encryptionkey@0"
I20260619 18:37:34.955842 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:34.967792 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:34.971024 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:34.971199 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:34.971771 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:34.972586 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:34.972633 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:34.972685 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:34.972711 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:34.981848 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43543
I20260619 18:37:34.981878 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43543 every 8 connection(s)
I20260619 18:37:34.982805 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:37:34.990060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 774
I20260619 18:37:34.990185 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:37:34.990490 392 external_mini_cluster.cc:1428] Setting key 6961d7d4bd75b9e5c0ad41a4941bba28
Jun 19 18:37:34 dist-test-slave-rxv6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:34.997387 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:34.984770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40323 (local address 127.0.98.62:42395)
0619 18:37:34.984930 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:34.984934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:34.986296 (+ 1362us) server_negotiation.cc:407] Connection header received
0619 18:37:34.987690 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:34.987693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:34.987760 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:34.987835 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:34.989039 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:34.989577 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:34.991113 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:34.991371 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:34.991465 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:34.993896 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:34.993899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:34.993901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:34.993955 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:34.995312 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:34.995845 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:34.995849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:34.995851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:34.995956 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:34.996289 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:34.996292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:34.996294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:34.996477 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:34.996564 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:34.997096 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 18:37:34.997216 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:37:34.998265 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:34.985184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42395 (local address 127.0.98.3:40323)
0619 18:37:34.986123 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:34.986170 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:34.987473 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:34.988020 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:34.988027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:34.988387 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:34.988898 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:34.988911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:34.989693 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:34.989695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:34.990991 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:34.991002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:34.991520 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:34.991524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:34.991706 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:34.992295 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:34.992311 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:34.993685 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:34.995413 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:34.995419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:34.995436 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:34.995749 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:34.996035 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:34.996038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:34.996041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:34.996180 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:34.996573 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:34.996581 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:34.996870 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 18:37:34.997041 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 18:37:34.997367 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":520,"threads_started":1}
I20260619 18:37:34.999473 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42395
I20260619 18:37:34.999722 903 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:35.000257 903 heartbeater.cc:511] Master 127.0.98.62:42395 requested a full tablet report, sending...
I20260619 18:37:35.001302 444 ts_manager.cc:194] Registered new tserver with Master: ab2fe314585743f0a6f3ada1580d4d47 (127.0.98.3:43543)
I20260619 18:37:35.001854 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40323
I20260619 18:37:35.005012 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:37:35 dist-test-slave-rxv6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894255, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:37:35 dist-test-slave-rxv6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894255, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 18:37:35.031776 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:35.021559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42006 (local address 127.0.98.1:45627)
0619 18:37:35.021818 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:35.021834 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:35.021865 (+ 31us) server_negotiation.cc:407] Connection header received
0619 18:37:35.022019 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:35.022026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:35.022240 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:35.022380 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:35.023040 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:35.023916 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:35.024826 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:35.025209 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:35.025312 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:35.027866 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:35.027875 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:35.027889 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:35.027916 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:35.029768 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:35.030287 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:35.030293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:35.030295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:35.030392 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:35.030831 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:35.030834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:35.030835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:35.031104 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:35.031273 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:35.031464 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 18:37:35.031607 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":60,"threads_started":1}
W20260619 18:37:35.032809 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42006
Jun 19 18:37:35 dist-test-slave-rxv6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:37:35 dist-test-slave-rxv6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 18:37:35.058159 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:35.049512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.0.98.1:45627)
0619 18:37:35.049677 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:35.049682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:35.049786 (+ 104us) server_negotiation.cc:407] Connection header received
0619 18:37:35.049940 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:35.049943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:35.049996 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:35.050068 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:35.050794 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:35.051300 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:35.052226 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:35.052466 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:35.052555 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:35.055082 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:35.055085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:35.055087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:35.055122 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:35.056487 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:35.056919 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:35.056922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:35.056923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:35.057003 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:35.057309 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:35.057311 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:35.057312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:35.057478 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:35.057574 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:35.057855 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 18:37:35.058005 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 18:37:35.059329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 506
I20260619 18:37:35.064841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 640
I20260619 18:37:35.071683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 774
I20260619 18:37:35.076288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 412
2026-06-19T18:37:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5204 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:37:35 dist-test-slave-rxv6 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:37:35 dist-test-slave-rxv6 krb5kdc[917](info): set up 2 sockets
Jun 19 18:37:35 dist-test-slave-rxv6 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260619 18:37:36.041452 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:37:36 dist-test-slave-rxv6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:37:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:37:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:37:36.157804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36447
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:36.264780 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:36.265074 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:36.265115 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:36.269207 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:37:36.269277 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:36.269296 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:36.269311 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:37:36.269325 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:37:36.274673 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:36.275686 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:36.276661 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:37:36.284932 930 server_base.cc:1034] running on GCE node
W20260619 18:37:36.284898 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:36.284896 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:36.284898 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:36.285657 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:36.286746 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:36.287926 930 hybrid_clock.cc:648] HybridClock initialized: now 1781894256287898 us; error 38 us; skew 500 ppm
Jun 19 18:37:36 dist-test-slave-rxv6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:36.290673 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:37:36.291572 930 webserver.cc:460] Webserver started at http://127.0.98.62:34635/ using document root <none> and password file <none>
I20260619 18:37:36.292115 930 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:36.292181 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:36.292366 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:36.294446 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "983f5d7b711f452c8d2af93369caa07e"
format_stamp: "Formatted at 2026-06-19 18:37:36 on dist-test-slave-rxv6"
server_key: "8022cc91982035f532f3bbfde824d325"
server_key_iv: "f121c968bce007ec5fdc6e65ed2aa96e"
server_key_version: "encryptionkey@0"
I20260619 18:37:36.294952 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "983f5d7b711f452c8d2af93369caa07e"
format_stamp: "Formatted at 2026-06-19 18:37:36 on dist-test-slave-rxv6"
server_key: "8022cc91982035f532f3bbfde824d325"
server_key_iv: "f121c968bce007ec5fdc6e65ed2aa96e"
server_key_version: "encryptionkey@0"
I20260619 18:37:36.298507 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:37:36.300782 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:36.302008 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:37:36.302122 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "983f5d7b711f452c8d2af93369caa07e"
format_stamp: "Formatted at 2026-06-19 18:37:36 on dist-test-slave-rxv6"
server_key: "8022cc91982035f532f3bbfde824d325"
server_key_iv: "f121c968bce007ec5fdc6e65ed2aa96e"
server_key_version: "encryptionkey@0"
I20260619 18:37:36.302207 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:36.329002 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:36.332139 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:36.332325 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:36.339509 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36447
I20260619 18:37:36.339545 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36447 every 8 connection(s)
I20260619 18:37:36.340474 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:37:36.343425 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:36.343849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 930
I20260619 18:37:36.343953 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:37:36.344185 392 external_mini_cluster.cc:1428] Setting key aa08e6bbb20a1fdf18d991d7c20ef90f
I20260619 18:37:36.350548 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e: Bootstrap starting.
I20260619 18:37:36.352846 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e: Neither blocks nor log segments found. Creating new log.
Jun 19 18:37:36 dist-test-slave-rxv6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:36.353750 998 log.cc:824] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:36.356163 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e: No bootstrap required, opened a new log
I20260619 18:37:36.359067 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:36.345582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.0.98.62:36447)
0619 18:37:36.346051 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:36.346070 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:36.346114 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:37:36.346831 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:36.346855 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:36.347291 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:36.347679 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:36.348309 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:36.349812 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:36.350731 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:36.351219 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:36.351366 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:36.353899 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:36.353928 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:36.353946 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:36.353977 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:36.355941 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:36.356458 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:36.356464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:36.356470 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:36.356611 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:36.356864 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:36.356867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:36.356869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:36.357359 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:36.357537 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:36.357749 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 18:37:36.358090 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":111,"threads_started":1}
I20260619 18:37:36.360352 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983f5d7b711f452c8d2af93369caa07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } }
I20260619 18:37:36.360580 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:36.360615 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983f5d7b711f452c8d2af93369caa07e, State: Initialized, Role: FOLLOWER
I20260619 18:37:36.361081 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983f5d7b711f452c8d2af93369caa07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } }
I20260619 18:37:36.361250 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:37:36.361316 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:37:36.361421 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:36.362325 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983f5d7b711f452c8d2af93369caa07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } }
I20260619 18:37:36.362646 998 leader_election.cc:302] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 983f5d7b711f452c8d2af93369caa07e; no voters:
I20260619 18:37:36.362898 998 leader_election.cc:288] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:37:36.363080 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:36.363274 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [term 1 LEADER]: Becoming Leader. State: Replica: 983f5d7b711f452c8d2af93369caa07e, State: Running, Role: LEADER
I20260619 18:37:36.363605 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983f5d7b711f452c8d2af93369caa07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } }
I20260619 18:37:36.364092 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:37:36.365175 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 983f5d7b711f452c8d2af93369caa07e. Latest consensus state: current_term: 1 leader_uuid: "983f5d7b711f452c8d2af93369caa07e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983f5d7b711f452c8d2af93369caa07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } } }
I20260619 18:37:36.365303 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:36.365685 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:37:36.366034 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "983f5d7b711f452c8d2af93369caa07e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "983f5d7b711f452c8d2af93369caa07e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } } }
I20260619 18:37:36.366156 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:36.368588 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:37:36.374106 1012 catalog_manager.cc:1347] Generated new cluster ID: aaec466140be4f109073a55189480e31
I20260619 18:37:36.374415 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:37:36.386188 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:37:36.387193 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:37:36.396373 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 983f5d7b711f452c8d2af93369caa07e: Generated new TSK 0
I20260619 18:37:36.397122 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:37:36.425302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:36.533226 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:36.533502 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:36.533546 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:36.537590 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:36.537667 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:36.537727 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:37:36.543144 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:36.544167 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:36.545123 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:36.554256 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:36.554366 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:36.554490 1024 server_base.cc:1034] running on GCE node
W20260619 18:37:36.554386 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:36.554963 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:36.555612 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:36.556788 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781894256556765 us; error 33 us; skew 500 ppm
Jun 19 18:37:36 dist-test-slave-rxv6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:36.559556 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:37:36.560412 1024 webserver.cc:460] Webserver started at http://127.0.98.1:40185/ using document root <none> and password file <none>
I20260619 18:37:36.560956 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:36.561014 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:36.561174 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:36.563171 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "77f0068ab5f041e0995826a0842610dc"
format_stamp: "Formatted at 2026-06-19 18:37:36 on dist-test-slave-rxv6"
server_key: "843def4d858e749a094ef2052a7a917a"
server_key_iv: "5882e6639a6afc03525373cdc164dff2"
server_key_version: "encryptionkey@0"
I20260619 18:37:36.563678 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "77f0068ab5f041e0995826a0842610dc"
format_stamp: "Formatted at 2026-06-19 18:37:36 on dist-test-slave-rxv6"
server_key: "843def4d858e749a094ef2052a7a917a"
server_key_iv: "5882e6639a6afc03525373cdc164dff2"
server_key_version: "encryptionkey@0"
I20260619 18:37:36.567528 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:37:36.570056 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:36.571309 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:37:36.571420 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "77f0068ab5f041e0995826a0842610dc"
format_stamp: "Formatted at 2026-06-19 18:37:36 on dist-test-slave-rxv6"
server_key: "843def4d858e749a094ef2052a7a917a"
server_key_iv: "5882e6639a6afc03525373cdc164dff2"
server_key_version: "encryptionkey@0"
I20260619 18:37:36.571516 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:36.601444 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:36.604947 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:36.605141 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:36.605739 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:36.606644 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:36.606693 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:36.606745 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:36.606774 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:36.615345 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37317
I20260619 18:37:36.615445 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37317 every 8 connection(s)
I20260619 18:37:36.616258 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:37:36.621603 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 1024
I20260619 18:37:36.621708 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:37:36.621999 392 external_mini_cluster.cc:1428] Setting key ae17c567afa45eb02364d82f0050bb50
Jun 19 18:37:36 dist-test-slave-rxv6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:36.630818 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:36.618118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41741 (local address 127.0.98.62:36447)
0619 18:37:36.618269 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:36.618274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:36.619124 (+ 850us) server_negotiation.cc:407] Connection header received
0619 18:37:36.620439 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:36.620442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:36.620494 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:36.620574 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:36.621877 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:36.622572 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:36.623838 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:36.624081 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:36.624169 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:36.626874 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:36.626878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:36.626881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:36.626910 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:36.628534 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:36.629025 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:36.629029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:36.629031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:36.629118 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:36.629416 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:36.629419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:36.629420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:36.629651 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:36.629755 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:36.630376 (+ 621us) server_negotiation.cc:299] Negotiation successful
0619 18:37:36.630559 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 18:37:36.631629 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:36.618534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36447 (local address 127.0.98.1:41741)
0619 18:37:36.618968 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:36.619012 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:36.620214 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:36.620735 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:36.620742 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:36.621119 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:36.621669 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:36.621685 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:36.622715 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:36.622718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:36.623727 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:36.623737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:36.624219 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:36.624223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:36.624398 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:36.624992 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:36.625014 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:36.626692 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:36.628651 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:36.628656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:36.628670 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:36.628923 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:36.629203 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:36.629205 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:36.629207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:36.629337 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:36.629741 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:36.629746 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:36.630057 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 18:37:36.630272 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 18:37:36.630651 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":117,"threads_started":1}
I20260619 18:37:36.632915 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36447
I20260619 18:37:36.633256 1153 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 18:37:36.633818 1153 heartbeater.cc:511] Master 127.0.98.62:36447 requested a full tablet report, sending...
I20260619 18:37:36.635607 962 ts_manager.cc:194] Registered new tserver with Master: 77f0068ab5f041e0995826a0842610dc (127.0.98.1:37317)
I20260619 18:37:36.636890 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:37:36.649701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:36.758497 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:36.758783 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:36.758843 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:36.762966 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:36.763049 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:36.763130 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:37:36.768630 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:36.769711 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:36.770769 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:36.780181 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:36.780221 1158 server_base.cc:1034] running on GCE node
W20260619 18:37:36.780236 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:36.780184 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:36.780939 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:36.781584 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:36.782744 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781894256782722 us; error 37 us; skew 500 ppm
Jun 19 18:37:36 dist-test-slave-rxv6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:36.785737 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:37:36.786638 1158 webserver.cc:460] Webserver started at http://127.0.98.2:35803/ using document root <none> and password file <none>
I20260619 18:37:36.787182 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:36.787245 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:36.787431 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:36.789481 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "718666829ca84d258a7fea1b7f9c0757"
format_stamp: "Formatted at 2026-06-19 18:37:36 on dist-test-slave-rxv6"
server_key: "9ac09519ae4bb0c12e3b046d575a15ea"
server_key_iv: "59493c0aaa166aedd27dc04c2977469e"
server_key_version: "encryptionkey@0"
I20260619 18:37:36.790053 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "718666829ca84d258a7fea1b7f9c0757"
format_stamp: "Formatted at 2026-06-19 18:37:36 on dist-test-slave-rxv6"
server_key: "9ac09519ae4bb0c12e3b046d575a15ea"
server_key_iv: "59493c0aaa166aedd27dc04c2977469e"
server_key_version: "encryptionkey@0"
I20260619 18:37:36.793774 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:37:36.796274 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:36.797492 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:37:36.797587 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "718666829ca84d258a7fea1b7f9c0757"
format_stamp: "Formatted at 2026-06-19 18:37:36 on dist-test-slave-rxv6"
server_key: "9ac09519ae4bb0c12e3b046d575a15ea"
server_key_iv: "59493c0aaa166aedd27dc04c2977469e"
server_key_version: "encryptionkey@0"
I20260619 18:37:36.797688 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:36.817582 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:36.820742 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:36.820930 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:36.821547 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:36.822374 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:36.822422 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:36.822459 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:36.822474 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:36.830639 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37819
I20260619 18:37:36.830672 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37819 every 8 connection(s)
I20260619 18:37:36.831503 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:37:36.835922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 1158
I20260619 18:37:36.836040 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:37:36.836273 392 external_mini_cluster.cc:1428] Setting key b0eabf3384619aeb04112e477d703fc0
Jun 19 18:37:36 dist-test-slave-rxv6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 18:37:36.845189 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:36.833327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49059 (local address 127.0.98.62:36447)
0619 18:37:36.833451 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:36.833454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:36.834368 (+ 914us) server_negotiation.cc:407] Connection header received
0619 18:37:36.835743 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:36.835747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:36.835803 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:36.835875 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:36.837193 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:36.837795 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:36.838990 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:36.839216 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:36.839305 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:36.841639 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:36.841642 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:36.841644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:36.841668 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:36.843126 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:36.843653 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:36.843656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:36.843658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:36.843736 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:36.844052 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:36.844055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:36.844057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:36.844250 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:36.844332 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:36.844849 (+ 517us) server_negotiation.cc:299] Negotiation successful
0619 18:37:36.844985 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:37:36.846065 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:36.833702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36447 (local address 127.0.98.2:49059)
0619 18:37:36.834223 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:36.834266 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:36.835510 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:36.836085 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:36.836095 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:36.836479 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:36.837054 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:36.837069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:36.837945 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:36.837949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:36.838871 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:36.838881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:36.839352 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:36.839356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:36.839514 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:36.840059 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:36.840075 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:36.841473 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:36.843255 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:36.843262 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:36.843280 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:36.843548 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:36.843820 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:36.843823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:36.843825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:36.843967 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:36.844345 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:36.844352 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:36.844615 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 18:37:36.844790 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 18:37:36.845131 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 18:37:36.847347 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36447
I20260619 18:37:36.847584 1287 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:36.848153 1287 heartbeater.cc:511] Master 127.0.98.62:36447 requested a full tablet report, sending...
I20260619 18:37:36.849262 962 ts_manager.cc:194] Registered new tserver with Master: 718666829ca84d258a7fea1b7f9c0757 (127.0.98.2:37819)
I20260619 18:37:36.849830 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:37:36.861395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:36.968920 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:36.969193 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:36.969236 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:36.973215 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:36.973280 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:36.973340 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:37:36.978720 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:36.979769 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:36.980758 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:36.990074 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:36.990123 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:36.990240 1292 server_base.cc:1034] running on GCE node
W20260619 18:37:36.990135 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:36.990707 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:36.991364 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:36.992523 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781894256992503 us; error 34 us; skew 500 ppm
Jun 19 18:37:36 dist-test-slave-rxv6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:36.995391 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:37:36.996233 1292 webserver.cc:460] Webserver started at http://127.0.98.3:35707/ using document root <none> and password file <none>
I20260619 18:37:36.996788 1292 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:36.996845 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:36.997056 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:36.999151 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "247e728815cf4361abc147c12bfad326"
format_stamp: "Formatted at 2026-06-19 18:37:36 on dist-test-slave-rxv6"
server_key: "bc62d7f9af241da388dd5eac8da548c7"
server_key_iv: "97faf13328ec2a3a0fe54b75d9be8780"
server_key_version: "encryptionkey@0"
I20260619 18:37:36.999677 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "247e728815cf4361abc147c12bfad326"
format_stamp: "Formatted at 2026-06-19 18:37:36 on dist-test-slave-rxv6"
server_key: "bc62d7f9af241da388dd5eac8da548c7"
server_key_iv: "97faf13328ec2a3a0fe54b75d9be8780"
server_key_version: "encryptionkey@0"
I20260619 18:37:37.003382 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:37:37.005812 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:37.007050 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:37:37.007190 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "247e728815cf4361abc147c12bfad326"
format_stamp: "Formatted at 2026-06-19 18:37:36 on dist-test-slave-rxv6"
server_key: "bc62d7f9af241da388dd5eac8da548c7"
server_key_iv: "97faf13328ec2a3a0fe54b75d9be8780"
server_key_version: "encryptionkey@0"
I20260619 18:37:37.007289 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:37.032323 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:37.035398 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:37.035583 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:37.036185 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:37.037003 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:37.037051 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:37.037093 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:37.037122 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:37.045555 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41565
I20260619 18:37:37.045567 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41565 every 8 connection(s)
I20260619 18:37:37.046468 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:37:37.047533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 1292
I20260619 18:37:37.047645 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:37:37.047930 392 external_mini_cluster.cc:1428] Setting key 9648fdd3850e3789a2f77486a78f62ed
Jun 19 18:37:37 dist-test-slave-rxv6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:37.060111 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:37.048359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48827 (local address 127.0.98.62:36447)
0619 18:37:37.048488 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:37.048493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:37.049451 (+ 958us) server_negotiation.cc:407] Connection header received
0619 18:37:37.050848 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:37.050851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:37.050903 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:37.050972 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:37.052128 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.052650 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.053900 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.054237 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.054343 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:37.056787 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:37.056790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:37.056792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:37.056817 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:37.058168 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:37.058636 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:37.058639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:37.058641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:37.058723 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:37.059089 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:37.059092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:37.059093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:37.059284 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:37.059368 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:37.059843 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 18:37:37.059948 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:37:37.061064 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:37.048812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36447 (local address 127.0.98.3:48827)
0619 18:37:37.049311 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:37.049364 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:37.050655 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:37.051154 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:37.051162 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:37.051507 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:37.051995 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:37.052009 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.052760 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.052764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:37.053734 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:37.053744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.054386 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.054390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:37.054575 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:37.055101 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:37.055117 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:37.056608 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:37.058282 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:37.058288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:37.058300 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:37.058552 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:37.058808 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:37.058810 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:37.058811 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:37.058941 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:37.059386 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:37.059391 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:37.059627 (+ 236us) client_negotiation.cc:769] Sending connection context
0619 18:37:37.059793 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 18:37:37.060095 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":128,"threads_started":1}
I20260619 18:37:37.062079 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36447
I20260619 18:37:37.062283 1421 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:37.062713 1421 heartbeater.cc:511] Master 127.0.98.62:36447 requested a full tablet report, sending...
I20260619 18:37:37.063618 962 ts_manager.cc:194] Registered new tserver with Master: 247e728815cf4361abc147c12bfad326 (127.0.98.3:41565)
I20260619 18:37:37.064147 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48827
I20260619 18:37:37.073392 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:37:37.083700 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:37.076806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56932 (local address 127.0.98.62:36447)
0619 18:37:37.076918 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:37.076923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:37.077088 (+ 165us) server_negotiation.cc:407] Connection header received
0619 18:37:37.077211 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:37.077215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:37.077256 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:37.077317 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:37.078023 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.078536 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.079330 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.079550 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.079633 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:37.080859 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:37.080863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:37.080865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:37.080887 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:37.082140 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:37.082546 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:37.082549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:37.082551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:37.082620 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:37.082865 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:37.082869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:37.082871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:37.083097 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:37.083213 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:37.083440 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 18:37:37.083552 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 18:37:37.088442 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:37:37.091539 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:37:37.106037 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:37.100761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37819 (local address 127.0.0.1:52520)
0619 18:37:37.101555 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:37.101567 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:37.101635 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:37.102427 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:37.102430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:37.102448 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:37.102534 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:37.102540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.103687 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.103691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:37.104619 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:37.104628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.105441 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.105445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:37.105547 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:37.105584 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 18:37:37.105718 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 18:37:37.105842 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260619 18:37:37.106120 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:37.101254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52520 (local address 127.0.98.2:37819)
0619 18:37:37.101998 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:37.102010 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:37.102029 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:37:37.102117 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:37.102122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:37.102291 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:37.102434 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:37.102639 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.103569 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.104758 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.105332 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.105429 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:37.105595 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:37.105836 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 18:37:37.105960 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":101,"threads_started":1}
I20260619 18:37:37.106353 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:37.100123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55962 (local address 127.0.98.1:37317)
0619 18:37:37.100398 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:37.100414 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:37.101117 (+ 703us) server_negotiation.cc:407] Connection header received
0619 18:37:37.101197 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:37.101202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:37.101400 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:37.101576 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:37.101998 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.103117 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.104638 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.105436 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.105516 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:37.105720 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:37.106107 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 18:37:37.106205 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":83,"threads_started":1}
I20260619 18:37:37.106499 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:37.100473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.0.98.3:41565)
0619 18:37:37.100845 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:37.100858 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:37.100975 (+ 117us) server_negotiation.cc:407] Connection header received
0619 18:37:37.101158 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:37.101164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:37.101375 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:37.101894 (+ 519us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:37.102073 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.102917 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.104626 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.105436 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.105543 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:37.105652 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:37.106269 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 18:37:37.106366 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":127,"threads_started":1}
I20260619 18:37:37.106539 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:37.099907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37317 (local address 127.0.0.1:55962)
0619 18:37:37.100751 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:37.100811 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:37.100950 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:37.101736 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:37.101739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:37.101756 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:37.101853 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:37.101859 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.103243 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.103247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:37.104489 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:37.104499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.105705 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.105708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:37.105826 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:37.105846 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 18:37:37.105933 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 18:37:37.105992 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":8,"thread_start_us":75,"threads_started":1}
I20260619 18:37:37.106184 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:37.100135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41565 (local address 127.0.0.1:49322)
0619 18:37:37.100869 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:37.100882 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:37.101035 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:37.101538 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:37.101544 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:37.101566 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:37.101716 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:37.101723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.103025 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.103028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:37.104489 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:37.104499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.105714 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.105718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:37.105863 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:37.105885 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:37:37.105982 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 18:37:37.106068 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":78,"thread_start_us":104,"threads_started":1}
I20260619 18:37:37.109123 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 527809140508462ea84756f125165b8e (DEFAULT_TABLE table=test-table [id=eebd06417537456cb4142b203adaa0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:37.109266 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 527809140508462ea84756f125165b8e (DEFAULT_TABLE table=test-table [id=eebd06417537456cb4142b203adaa0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:37.109266 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 527809140508462ea84756f125165b8e (DEFAULT_TABLE table=test-table [id=eebd06417537456cb4142b203adaa0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:37.110169 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527809140508462ea84756f125165b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:37.110291 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527809140508462ea84756f125165b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:37.110291 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527809140508462ea84756f125165b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:37.116237 1438 tablet_bootstrap.cc:492] T 527809140508462ea84756f125165b8e P 77f0068ab5f041e0995826a0842610dc: Bootstrap starting.
I20260619 18:37:37.117823 1439 tablet_bootstrap.cc:492] T 527809140508462ea84756f125165b8e P 247e728815cf4361abc147c12bfad326: Bootstrap starting.
I20260619 18:37:37.118309 1440 tablet_bootstrap.cc:492] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757: Bootstrap starting.
I20260619 18:37:37.118925 1438 tablet_bootstrap.cc:654] T 527809140508462ea84756f125165b8e P 77f0068ab5f041e0995826a0842610dc: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:37.119925 1440 tablet_bootstrap.cc:654] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:37.119921 1439 tablet_bootstrap.cc:654] T 527809140508462ea84756f125165b8e P 247e728815cf4361abc147c12bfad326: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:37.120220 1438 log.cc:824] T 527809140508462ea84756f125165b8e P 77f0068ab5f041e0995826a0842610dc: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:37.121022 1440 log.cc:824] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:37.121026 1439 log.cc:824] T 527809140508462ea84756f125165b8e P 247e728815cf4361abc147c12bfad326: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:37.122416 1438 tablet_bootstrap.cc:492] T 527809140508462ea84756f125165b8e P 77f0068ab5f041e0995826a0842610dc: No bootstrap required, opened a new log
I20260619 18:37:37.122617 1438 ts_tablet_manager.cc:1397] T 527809140508462ea84756f125165b8e P 77f0068ab5f041e0995826a0842610dc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 18:37:37.122859 1439 tablet_bootstrap.cc:492] T 527809140508462ea84756f125165b8e P 247e728815cf4361abc147c12bfad326: No bootstrap required, opened a new log
I20260619 18:37:37.123023 1439 ts_tablet_manager.cc:1397] T 527809140508462ea84756f125165b8e P 247e728815cf4361abc147c12bfad326: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:37:37.123448 1440 tablet_bootstrap.cc:492] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757: No bootstrap required, opened a new log
I20260619 18:37:37.123636 1440 ts_tablet_manager.cc:1397] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:37:37.125756 1439 raft_consensus.cc:348] T 527809140508462ea84756f125165b8e P 247e728815cf4361abc147c12bfad326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f0068ab5f041e0995826a0842610dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37317 } } peers { permanent_uuid: "718666829ca84d258a7fea1b7f9c0757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37819 } } peers { permanent_uuid: "247e728815cf4361abc147c12bfad326" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41565 } }
I20260619 18:37:37.126011 1439 raft_consensus.cc:374] T 527809140508462ea84756f125165b8e P 247e728815cf4361abc147c12bfad326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:37.126097 1439 raft_consensus.cc:729] T 527809140508462ea84756f125165b8e P 247e728815cf4361abc147c12bfad326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247e728815cf4361abc147c12bfad326, State: Initialized, Role: FOLLOWER
I20260619 18:37:37.126592 1439 consensus_queue.cc:260] T 527809140508462ea84756f125165b8e P 247e728815cf4361abc147c12bfad326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f0068ab5f041e0995826a0842610dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37317 } } peers { permanent_uuid: "718666829ca84d258a7fea1b7f9c0757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37819 } } peers { permanent_uuid: "247e728815cf4361abc147c12bfad326" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41565 } }
I20260619 18:37:37.126518 1438 raft_consensus.cc:348] T 527809140508462ea84756f125165b8e P 77f0068ab5f041e0995826a0842610dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f0068ab5f041e0995826a0842610dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37317 } } peers { permanent_uuid: "718666829ca84d258a7fea1b7f9c0757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37819 } } peers { permanent_uuid: "247e728815cf4361abc147c12bfad326" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41565 } }
I20260619 18:37:37.126798 1438 raft_consensus.cc:374] T 527809140508462ea84756f125165b8e P 77f0068ab5f041e0995826a0842610dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:37.126901 1438 raft_consensus.cc:729] T 527809140508462ea84756f125165b8e P 77f0068ab5f041e0995826a0842610dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77f0068ab5f041e0995826a0842610dc, State: Initialized, Role: FOLLOWER
I20260619 18:37:37.127398 1421 heartbeater.cc:503] Master 127.0.98.62:36447 was elected leader, sending a full tablet report...
I20260619 18:37:37.127661 1439 ts_tablet_manager.cc:1428] T 527809140508462ea84756f125165b8e P 247e728815cf4361abc147c12bfad326: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 18:37:37.127737 1440 raft_consensus.cc:348] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f0068ab5f041e0995826a0842610dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37317 } } peers { permanent_uuid: "718666829ca84d258a7fea1b7f9c0757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37819 } } peers { permanent_uuid: "247e728815cf4361abc147c12bfad326" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41565 } }
I20260619 18:37:37.128001 1440 raft_consensus.cc:374] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:37.128057 1440 raft_consensus.cc:729] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718666829ca84d258a7fea1b7f9c0757, State: Initialized, Role: FOLLOWER
I20260619 18:37:37.128476 1440 consensus_queue.cc:260] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f0068ab5f041e0995826a0842610dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37317 } } peers { permanent_uuid: "718666829ca84d258a7fea1b7f9c0757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37819 } } peers { permanent_uuid: "247e728815cf4361abc147c12bfad326" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41565 } }
I20260619 18:37:37.128667 1438 consensus_queue.cc:260] T 527809140508462ea84756f125165b8e P 77f0068ab5f041e0995826a0842610dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f0068ab5f041e0995826a0842610dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37317 } } peers { permanent_uuid: "718666829ca84d258a7fea1b7f9c0757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37819 } } peers { permanent_uuid: "247e728815cf4361abc147c12bfad326" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41565 } }
I20260619 18:37:37.129115 1287 heartbeater.cc:503] Master 127.0.98.62:36447 was elected leader, sending a full tablet report...
I20260619 18:37:37.129472 1440 ts_tablet_manager.cc:1428] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 18:37:37.129566 1438 ts_tablet_manager.cc:1428] T 527809140508462ea84756f125165b8e P 77f0068ab5f041e0995826a0842610dc: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 18:37:37.129875 1153 heartbeater.cc:503] Master 127.0.98.62:36447 was elected leader, sending a full tablet report...
I20260619 18:37:37.177765 1445 raft_consensus.cc:482] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:37:37.177968 1445 raft_consensus.cc:504] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f0068ab5f041e0995826a0842610dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37317 } } peers { permanent_uuid: "718666829ca84d258a7fea1b7f9c0757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37819 } } peers { permanent_uuid: "247e728815cf4361abc147c12bfad326" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41565 } }
I20260619 18:37:37.179171 1445 leader_election.cc:288] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77f0068ab5f041e0995826a0842610dc (127.0.98.1:37317), 247e728815cf4361abc147c12bfad326 (127.0.98.3:41565)
I20260619 18:37:37.183369 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:37.179521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35853 (local address 127.0.98.3:41565)
0619 18:37:37.179634 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:37.179638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:37.179653 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:37:37.179711 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:37.179713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:37.179751 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:37.179821 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:37.180134 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.180790 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.182260 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.182875 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.182970 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:37.183035 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:37.183212 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 18:37:37.183266 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:37:37.183393 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:37.179410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41565 (local address 127.0.98.2:35853)
0619 18:37:37.179535 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:37.179551 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:37.179632 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:37.179859 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:37.179862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:37.179881 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:37.179987 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:37.179992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.180905 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.180909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:37.182150 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:37.182159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.182987 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.182991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:37.183097 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:37.183122 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:37:37.183200 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 18:37:37.183263 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 18:37:37.184067 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:37.179471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50565 (local address 127.0.98.1:37317)
0619 18:37:37.179614 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:37.179619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:37.180337 (+ 718us) server_negotiation.cc:407] Connection header received
0619 18:37:37.180397 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:37.180400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:37.180451 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:37.180524 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:37.180745 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.181490 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.182988 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.183596 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.183701 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:37.183757 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:37.183917 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 18:37:37.183973 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:37:37.184093 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:37.179413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37317 (local address 127.0.98.2:50565)
0619 18:37:37.180247 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:37.180258 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:37.180336 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:37.180547 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:37.180549 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:37.180559 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:37.180658 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:37.180662 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.181618 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.181621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:37.182849 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:37.182860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.183731 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.183735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:37.183806 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:37.183828 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:37:37.183904 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 18:37:37.183963 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":468,"thread_start_us":157,"threads_started":1}
I20260619 18:37:37.184137 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527809140508462ea84756f125165b8e" candidate_uuid: "718666829ca84d258a7fea1b7f9c0757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247e728815cf4361abc147c12bfad326" is_pre_election: true
I20260619 18:37:37.184463 1375 raft_consensus.cc:2454] T 527809140508462ea84756f125165b8e P 247e728815cf4361abc147c12bfad326 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 718666829ca84d258a7fea1b7f9c0757 in term 0.
I20260619 18:37:37.184681 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527809140508462ea84756f125165b8e" candidate_uuid: "718666829ca84d258a7fea1b7f9c0757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77f0068ab5f041e0995826a0842610dc" is_pre_election: true
I20260619 18:37:37.184975 1107 raft_consensus.cc:2454] T 527809140508462ea84756f125165b8e P 77f0068ab5f041e0995826a0842610dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 718666829ca84d258a7fea1b7f9c0757 in term 0.
I20260619 18:37:37.184976 1174 leader_election.cc:302] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247e728815cf4361abc147c12bfad326, 718666829ca84d258a7fea1b7f9c0757; no voters:
I20260619 18:37:37.185276 1445 raft_consensus.cc:2781] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:37:37.185372 1445 raft_consensus.cc:482] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:37:37.185427 1445 raft_consensus.cc:3037] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:37.186605 1445 raft_consensus.cc:504] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f0068ab5f041e0995826a0842610dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37317 } } peers { permanent_uuid: "718666829ca84d258a7fea1b7f9c0757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37819 } } peers { permanent_uuid: "247e728815cf4361abc147c12bfad326" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41565 } }
I20260619 18:37:37.186944 1445 leader_election.cc:288] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 [CANDIDATE]: Term 1 election: Requested vote from peers 77f0068ab5f041e0995826a0842610dc (127.0.98.1:37317), 247e728815cf4361abc147c12bfad326 (127.0.98.3:41565)
I20260619 18:37:37.187282 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527809140508462ea84756f125165b8e" candidate_uuid: "718666829ca84d258a7fea1b7f9c0757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77f0068ab5f041e0995826a0842610dc"
I20260619 18:37:37.187321 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527809140508462ea84756f125165b8e" candidate_uuid: "718666829ca84d258a7fea1b7f9c0757" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247e728815cf4361abc147c12bfad326"
I20260619 18:37:37.187423 1375 raft_consensus.cc:3037] T 527809140508462ea84756f125165b8e P 247e728815cf4361abc147c12bfad326 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:37.187414 1107 raft_consensus.cc:3037] T 527809140508462ea84756f125165b8e P 77f0068ab5f041e0995826a0842610dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:37.188638 1375 raft_consensus.cc:2454] T 527809140508462ea84756f125165b8e P 247e728815cf4361abc147c12bfad326 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 718666829ca84d258a7fea1b7f9c0757 in term 1.
I20260619 18:37:37.188647 1107 raft_consensus.cc:2454] T 527809140508462ea84756f125165b8e P 77f0068ab5f041e0995826a0842610dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 718666829ca84d258a7fea1b7f9c0757 in term 1.
I20260619 18:37:37.188961 1176 leader_election.cc:302] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 718666829ca84d258a7fea1b7f9c0757, 77f0068ab5f041e0995826a0842610dc; no voters:
I20260619 18:37:37.189127 1445 raft_consensus.cc:2781] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:37.189332 1445 raft_consensus.cc:686] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 [term 1 LEADER]: Becoming Leader. State: Replica: 718666829ca84d258a7fea1b7f9c0757, State: Running, Role: LEADER
I20260619 18:37:37.189695 1445 consensus_queue.cc:237] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f0068ab5f041e0995826a0842610dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37317 } } peers { permanent_uuid: "718666829ca84d258a7fea1b7f9c0757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37819 } } peers { permanent_uuid: "247e728815cf4361abc147c12bfad326" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41565 } }
I20260619 18:37:37.192812 961 catalog_manager.cc:5495] T 527809140508462ea84756f125165b8e P 718666829ca84d258a7fea1b7f9c0757 reported cstate change: term changed from 0 to 1, leader changed from <none> to 718666829ca84d258a7fea1b7f9c0757 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "718666829ca84d258a7fea1b7f9c0757" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77f0068ab5f041e0995826a0842610dc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "718666829ca84d258a7fea1b7f9c0757" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "247e728815cf4361abc147c12bfad326" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41565 } health_report { overall_health: UNKNOWN } } }
Jun 19 18:37:37 dist-test-slave-rxv6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 18:37:37.216650 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:37.207956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.0.98.1:37317)
0619 18:37:37.208120 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:37.208125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:37.208279 (+ 154us) server_negotiation.cc:407] Connection header received
0619 18:37:37.208410 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:37.208414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:37.208457 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:37.208524 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:37.209196 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.209733 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.210647 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.210896 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.210985 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:37.213432 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:37.213447 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:37.213464 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:37.213488 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:37.215120 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:37.215498 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:37.215504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:37.215505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:37.215583 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:37.215857 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:37.215859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:37.215860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:37.216101 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:37.216250 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:37.216417 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 18:37:37.216511 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 18:37:37 dist-test-slave-rxv6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894257, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:37:37 dist-test-slave-rxv6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894257, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 18:37:37.241472 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:37.233443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55974 (local address 127.0.98.1:37317)
0619 18:37:37.233623 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:37.233627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:37.233689 (+ 62us) server_negotiation.cc:407] Connection header received
0619 18:37:37.233828 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:37.233830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:37.233871 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:37.233976 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:37.234577 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.235108 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.235855 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.236078 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.236152 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:37.238322 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:37.238326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:37.238329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:37.238352 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:37.239761 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:37.240230 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:37.240233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:37.240235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:37.240340 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:37.240638 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:37.240642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:37.240644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:37.240839 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:37.240939 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:37.241215 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 18:37:37.241315 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 18:37:37 dist-test-slave-rxv6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:37:37 dist-test-slave-rxv6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 18:37:37.266621 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:37.258246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55988 (local address 127.0.98.1:37317)
0619 18:37:37.258406 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:37.258410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:37.258503 (+ 93us) server_negotiation.cc:407] Connection header received
0619 18:37:37.258643 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:37.258646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:37.258691 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:37.258754 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:37.259508 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.260028 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.260832 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:37.261048 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:37.261120 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:37.263488 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:37.263491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:37.263493 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:37.263516 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:37.264756 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:37.265213 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:37.265216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:37.265217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:37.265303 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:37.265566 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:37.265569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:37.265571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:37.265779 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:37.265879 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:37.266362 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 18:37:37.266466 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 18:37:37.268023 1067 tablet_service.cc:2908] Rejecting scan request for tablet 527809140508462ea84756f125165b8e: Uninitialized: clean time has not yet been initialized
I20260619 18:37:37.270419 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 1024
I20260619 18:37:37.275632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 1158
I20260619 18:37:37.280443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 1292
I20260619 18:37:37.284916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 930
2026-06-19T18:37:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2208 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:37:37 dist-test-slave-rxv6 krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:37:37 dist-test-slave-rxv6 krb5kdc[1462](info): set up 2 sockets
Jun 19 18:37:37 dist-test-slave-rxv6 krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260619 18:37:38.245539 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:37:38 dist-test-slave-rxv6 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:37:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:37:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:37:38.365571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38207
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:38.473748 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:38.474112 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:38.474188 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:38.478255 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:37:38.478322 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 18:37:38.478344 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:38.478366 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:38.478389 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:37:38.478417 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:37:38.483800 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38207
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:38.484932 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:38.485932 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:38.493870 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:38.493969 1475 server_base.cc:1034] running on GCE node
W20260619 18:37:38.493865 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:38.493939 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:38.494773 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:38.495877 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:38.497042 1475 hybrid_clock.cc:648] HybridClock initialized: now 1781894258497016 us; error 41 us; skew 500 ppm
Jun 19 18:37:38 dist-test-slave-rxv6 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:38.499719 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:37:38.500588 1475 webserver.cc:460] Webserver started at http://127.0.98.62:33077/ using document root <none> and password file <none>
I20260619 18:37:38.501132 1475 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:38.501228 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:38.501421 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:38.503448 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "f499e5cf4e084db891ab671c97a8d063"
format_stamp: "Formatted at 2026-06-19 18:37:38 on dist-test-slave-rxv6"
server_key: "2653bd42b44b795ab4ad23850953b076"
server_key_iv: "d3ed041c8d9be29912cc7f778ed7e284"
server_key_version: "encryptionkey@0"
I20260619 18:37:38.503983 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "f499e5cf4e084db891ab671c97a8d063"
format_stamp: "Formatted at 2026-06-19 18:37:38 on dist-test-slave-rxv6"
server_key: "2653bd42b44b795ab4ad23850953b076"
server_key_iv: "d3ed041c8d9be29912cc7f778ed7e284"
server_key_version: "encryptionkey@0"
I20260619 18:37:38.507761 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 18:37:38.510121 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:38.511404 1475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:37:38.511533 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "f499e5cf4e084db891ab671c97a8d063"
format_stamp: "Formatted at 2026-06-19 18:37:38 on dist-test-slave-rxv6"
server_key: "2653bd42b44b795ab4ad23850953b076"
server_key_iv: "d3ed041c8d9be29912cc7f778ed7e284"
server_key_version: "encryptionkey@0"
I20260619 18:37:38.511626 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:38.527783 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:38.530722 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:38.530900 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:38.538393 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38207
I20260619 18:37:38.538429 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38207 every 8 connection(s)
I20260619 18:37:38.539348 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:37:38.541489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 1475
I20260619 18:37:38.541592 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:37:38.541874 392 external_mini_cluster.cc:1428] Setting key 0c7997689e6153709e8709af23799a5c
I20260619 18:37:38.542430 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:38.548761 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063: Bootstrap starting.
Jun 19 18:37:38 dist-test-slave-rxv6 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:38.551463 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:38.552628 1553 log.cc:824] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:38.554792 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063: No bootstrap required, opened a new log
I20260619 18:37:38.556113 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:38.543213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35566 (local address 127.0.98.62:38207)
0619 18:37:38.543594 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:38.543610 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:38.543644 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:37:38.544284 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:38.544302 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:38.544624 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:38.544898 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:38.545711 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:38.546780 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:38.547924 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:38.548389 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:38.548503 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:38.551352 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:38.551357 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:38.551371 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:38.551399 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:38.553164 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:38.553641 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:38.553648 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:38.553653 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:38.553783 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:38.554083 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:38.554085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:38.554087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:38.554529 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:38.554677 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:38.555067 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 18:37:38.555393 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
I20260619 18:37:38.558171 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f499e5cf4e084db891ab671c97a8d063" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } }
I20260619 18:37:38.558398 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:38.558455 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f499e5cf4e084db891ab671c97a8d063, State: Initialized, Role: FOLLOWER
I20260619 18:37:38.558877 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f499e5cf4e084db891ab671c97a8d063" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } }
I20260619 18:37:38.559005 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:37:38.559046 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:37:38.559139 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:38.560029 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f499e5cf4e084db891ab671c97a8d063" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } }
I20260619 18:37:38.560352 1553 leader_election.cc:302] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f499e5cf4e084db891ab671c97a8d063; no voters:
I20260619 18:37:38.560686 1553 leader_election.cc:288] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:37:38.560758 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:38.560959 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [term 1 LEADER]: Becoming Leader. State: Replica: f499e5cf4e084db891ab671c97a8d063, State: Running, Role: LEADER
I20260619 18:37:38.561342 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f499e5cf4e084db891ab671c97a8d063" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } }
I20260619 18:37:38.561816 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:37:38.562966 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f499e5cf4e084db891ab671c97a8d063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f499e5cf4e084db891ab671c97a8d063" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } } }
I20260619 18:37:38.563069 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:38.563298 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f499e5cf4e084db891ab671c97a8d063. Latest consensus state: current_term: 1 leader_uuid: "f499e5cf4e084db891ab671c97a8d063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f499e5cf4e084db891ab671c97a8d063" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38207 } } }
I20260619 18:37:38.563434 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063 [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:38.563454 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:37:38.566540 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:37:38.572397 1567 catalog_manager.cc:1347] Generated new cluster ID: 7f538a53375e4727b3c48304e6c738ba
I20260619 18:37:38.572465 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:37:38.584365 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:37:38.585296 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:37:38.589603 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063: Generated new TSK 0
I20260619 18:37:38.590436 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:37:38.624089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38207
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:38.731365 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:38.731633 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:38.731678 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:38.735674 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:38.735743 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:38.735782 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:37:38.735829 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:37:38.741086 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:38.742221 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:38.743242 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:38.752708 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:38.752725 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:38.752774 1579 server_base.cc:1034] running on GCE node
W20260619 18:37:38.752736 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:38.753394 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:38.754043 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:38.755198 1579 hybrid_clock.cc:648] HybridClock initialized: now 1781894258755159 us; error 51 us; skew 500 ppm
Jun 19 18:37:38 dist-test-slave-rxv6 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:38.757875 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:37:38.758738 1579 webserver.cc:460] Webserver started at http://127.0.98.1:34117/ using document root <none> and password file <none>
I20260619 18:37:38.759274 1579 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:38.759400 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:38.759601 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:38.761636 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "7752b495734f4010920a0ffec22d73b9"
format_stamp: "Formatted at 2026-06-19 18:37:38 on dist-test-slave-rxv6"
server_key: "0bdebd13021628feb526bb34731bc384"
server_key_iv: "6d528acd406f26129d5f3eb1659469e9"
server_key_version: "encryptionkey@0"
I20260619 18:37:38.762274 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7752b495734f4010920a0ffec22d73b9"
format_stamp: "Formatted at 2026-06-19 18:37:38 on dist-test-slave-rxv6"
server_key: "0bdebd13021628feb526bb34731bc384"
server_key_iv: "6d528acd406f26129d5f3eb1659469e9"
server_key_version: "encryptionkey@0"
I20260619 18:37:38.766079 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:37:38.768412 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:38.769586 1579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:37:38.769680 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "7752b495734f4010920a0ffec22d73b9"
format_stamp: "Formatted at 2026-06-19 18:37:38 on dist-test-slave-rxv6"
server_key: "0bdebd13021628feb526bb34731bc384"
server_key_iv: "6d528acd406f26129d5f3eb1659469e9"
server_key_version: "encryptionkey@0"
I20260619 18:37:38.769763 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:38.787266 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:38.790432 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:38.790606 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:38.792027 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:38.792076 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:38.792117 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:38.792145 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:37:38 dist-test-slave-rxv6 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:38.802273 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35651
I20260619 18:37:38.802335 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35651 every 8 connection(s)
I20260619 18:37:38.803232 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:37:38.805076 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:38.792901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42635 (local address 127.0.98.62:38207)
0619 18:37:38.793023 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:38.793026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:38.794141 (+ 1115us) server_negotiation.cc:407] Connection header received
0619 18:37:38.795240 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:38.795243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:38.795298 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:38.795381 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:38.796573 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:38.797099 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:38.798315 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:38.798623 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:38.798715 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:38.801597 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:38.801600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:38.801602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:38.801627 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:38.803086 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:38.803639 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:38.803644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:38.803645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:38.803735 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:38.804102 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:38.804111 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:38.804113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:38.804346 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:38.804428 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:38.804856 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 18:37:38.804958 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:37:38.805892 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:38.793444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38207 (local address 127.0.98.1:42635)
0619 18:37:38.793991 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:38.794038 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:38.795025 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:38.795556 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:38.795562 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:38.795886 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:38.796439 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:38.796452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:38.797213 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:38.797216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:38.798194 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:38.798204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:38.798787 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:38.798792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:38.798961 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:38.799553 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:38.799571 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:38.801437 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:38.803197 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:38.803202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:38.803215 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:38.803547 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:38.803842 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:38.803845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:38.803847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:38.804005 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:38.804434 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:38.804439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:38.804654 (+ 215us) client_negotiation.cc:769] Sending connection context
0619 18:37:38.804825 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 18:37:38.805063 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":134,"threads_started":1}
I20260619 18:37:38.807049 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38207
I20260619 18:37:38.807179 1710 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:38.807636 1710 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
I20260619 18:37:38.809159 1506 ts_manager.cc:194] Registered new tserver with Master: 7752b495734f4010920a0ffec22d73b9 (127.0.98.1:35651)
I20260619 18:37:38.810398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 1579
I20260619 18:37:38.810451 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42635
I20260619 18:37:38.810510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:37:38.810775 392 external_mini_cluster.cc:1428] Setting key 21f49739283c02d49f0c911e5931e9ae
I20260619 18:37:38.816148 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:38.808989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38207 (local address 127.0.98.1:44645)
0619 18:37:38.809240 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:38.809255 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:38.809348 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:38.809540 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:38.809543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:38.809727 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:38.809848 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:38.809856 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:38.810793 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:38.810797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:38.811553 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:38.811563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:38.812131 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:38.812135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:38.812218 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:38.812657 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:38.812670 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:38.813050 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:38.814779 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:38.814781 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:38.814783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:38.814985 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:38.815273 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:38.815275 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:38.815276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:38.815349 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:38.815688 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:38.815691 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:38.815762 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 18:37:38.815877 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 18:37:38.816001 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":80,"threads_started":1}
I20260619 18:37:38.816255 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:38.809059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44645 (local address 127.0.98.62:38207)
0619 18:37:38.809174 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:38.809178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:38.809311 (+ 133us) server_negotiation.cc:407] Connection header received
0619 18:37:38.809415 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:38.809417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:38.809451 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:38.809510 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:38.809966 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:38.810650 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:38.811665 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:38.812009 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:38.812101 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:38.813180 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:38.813183 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:38.813185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:38.813204 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:38.814668 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:38.815099 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:38.815103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:38.815104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:38.815193 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:38.815417 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:38.815419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:38.815421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:38.815604 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:38.815694 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:38.815945 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 18:37:38.816058 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:37:38.834831 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38207
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:38.944487 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:38.944748 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:38.944790 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:38.948802 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:38.948863 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:38.948906 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:37:38.948942 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:37:38.954113 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:38.955116 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:38.956090 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:38.965495 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:38.965495 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:38.965505 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:38.965926 1719 server_base.cc:1034] running on GCE node
I20260619 18:37:38.966302 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:38.966933 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:38.968088 1719 hybrid_clock.cc:648] HybridClock initialized: now 1781894258968073 us; error 32 us; skew 500 ppm
Jun 19 18:37:38 dist-test-slave-rxv6 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:38.970788 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:37:38.971642 1719 webserver.cc:460] Webserver started at http://127.0.98.2:37309/ using document root <none> and password file <none>
I20260619 18:37:38.972177 1719 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:38.972240 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:38.972421 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:38.974494 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "e2fb3b4f805341499b148957933cc70b"
format_stamp: "Formatted at 2026-06-19 18:37:38 on dist-test-slave-rxv6"
server_key: "5ab3ecab5708769c5dc2e5509723be3f"
server_key_iv: "06c671a914f8dccc2e4734756408feb9"
server_key_version: "encryptionkey@0"
I20260619 18:37:38.975020 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e2fb3b4f805341499b148957933cc70b"
format_stamp: "Formatted at 2026-06-19 18:37:38 on dist-test-slave-rxv6"
server_key: "5ab3ecab5708769c5dc2e5509723be3f"
server_key_iv: "06c671a914f8dccc2e4734756408feb9"
server_key_version: "encryptionkey@0"
I20260619 18:37:38.978531 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:37:38.980852 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:38.982196 1719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:37:38.982302 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "e2fb3b4f805341499b148957933cc70b"
format_stamp: "Formatted at 2026-06-19 18:37:38 on dist-test-slave-rxv6"
server_key: "5ab3ecab5708769c5dc2e5509723be3f"
server_key_iv: "06c671a914f8dccc2e4734756408feb9"
server_key_version: "encryptionkey@0"
I20260619 18:37:38.982396 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:39.001551 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:39.004818 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:39.005009 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:39.006565 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:39.006618 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:39.006659 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:39.006686 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:39.016692 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34839
Jun 19 18:37:39 dist-test-slave-rxv6 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:39.016741 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34839 every 8 connection(s)
I20260619 18:37:39.017658 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:37:39.020820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 1719
I20260619 18:37:39.020924 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:37:39.020961 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.007395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56655 (local address 127.0.98.62:38207)
0619 18:37:39.007508 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:39.007511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:39.008644 (+ 1133us) server_negotiation.cc:407] Connection header received
0619 18:37:39.009734 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:39.009738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:39.009798 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:39.009852 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:39.011603 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.012394 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.014105 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.014319 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.014413 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.017358 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:39.017362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:39.017365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:39.017395 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:39.019001 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:39.019479 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:39.019483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:39.019485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:39.019571 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:39.019882 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:39.019886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:39.019888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:39.020078 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:39.020163 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:39.020655 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 18:37:39.020786 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:37:39.021225 392 external_mini_cluster.cc:1428] Setting key 7099c6817d225cb677e8cf7abd099415
I20260619 18:37:39.021819 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.007929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38207 (local address 127.0.98.2:56655)
0619 18:37:39.008493 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:39.008536 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:39.009488 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:39.010317 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:39.010335 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:39.010734 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:39.011455 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.011477 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.012521 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.012525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.013983 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.013995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.014481 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.014485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.014695 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.015450 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:39.015472 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:39.017185 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:39.019108 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:39.019114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:39.019130 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:39.019387 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:39.019653 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:39.019656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:39.019658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:39.019792 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:39.020172 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:39.020179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:39.020434 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 18:37:39.020599 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 18:37:39.020921 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":138,"threads_started":1}
I20260619 18:37:39.023110 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38207
I20260619 18:37:39.023272 1850 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:39.023847 1850 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
I20260619 18:37:39.024896 1506 ts_manager.cc:194] Registered new tserver with Master: e2fb3b4f805341499b148957933cc70b (127.0.98.2:34839)
I20260619 18:37:39.025576 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56655
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 18:37:39.033324 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.025392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38207 (local address 127.0.98.2:55409)
0619 18:37:39.025665 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:39.025680 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:39.025775 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:39.026021 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:39.026024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:39.026224 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:39.026354 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.026361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.027342 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.027346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.028213 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.028224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.028953 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.028984 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.029107 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.029675 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:39.029689 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:39.030195 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:39.031878 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:39.031881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:39.031882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:39.032080 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:39.032427 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:39.032429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:39.032431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:39.032506 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:39.032858 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:39.032861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:39.032938 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 18:37:39.033016 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 18:37:39.033153 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":106,"threads_started":1}
I20260619 18:37:39.033380 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.025326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55409 (local address 127.0.98.62:38207)
0619 18:37:39.025440 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:39.025445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:39.025737 (+ 292us) server_negotiation.cc:407] Connection header received
0619 18:37:39.025864 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:39.025867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:39.025936 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:39.026005 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:39.026467 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.027185 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.028365 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.028821 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.028933 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.030375 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:39.030379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:39.030382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:39.030408 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:39.031765 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:39.032188 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:39.032192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:39.032194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:39.032338 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:39.032569 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:39.032571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:39.032573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:39.032780 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:39.032857 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:39.033120 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 18:37:39.033236 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:37:39.049396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38207
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:39.160919 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:39.161197 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:39.161262 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:39.165584 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:39.165647 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:39.165685 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:37:39.165719 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:37:39.171007 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:39.172003 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:39.173017 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:39.182333 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:39.182335 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:39.182469 1859 server_base.cc:1034] running on GCE node
W20260619 18:37:39.182335 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:39.183018 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:39.183684 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:39.184839 1859 hybrid_clock.cc:648] HybridClock initialized: now 1781894259184818 us; error 34 us; skew 500 ppm
Jun 19 18:37:39 dist-test-slave-rxv6 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:39.187537 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:37:39.188388 1859 webserver.cc:460] Webserver started at http://127.0.98.3:41145/ using document root <none> and password file <none>
I20260619 18:37:39.188928 1859 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:39.188993 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:39.189179 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:39.191231 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "8dc04b1c3b914c20981c740d78926d06"
format_stamp: "Formatted at 2026-06-19 18:37:39 on dist-test-slave-rxv6"
server_key: "76954dacb2da43d3da7a124a03d40688"
server_key_iv: "9d1857093ad7a06e24e29903822e4129"
server_key_version: "encryptionkey@0"
I20260619 18:37:39.191792 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8dc04b1c3b914c20981c740d78926d06"
format_stamp: "Formatted at 2026-06-19 18:37:39 on dist-test-slave-rxv6"
server_key: "76954dacb2da43d3da7a124a03d40688"
server_key_iv: "9d1857093ad7a06e24e29903822e4129"
server_key_version: "encryptionkey@0"
I20260619 18:37:39.195663 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:37:39.198356 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:39.199700 1859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:37:39.199817 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "8dc04b1c3b914c20981c740d78926d06"
format_stamp: "Formatted at 2026-06-19 18:37:39 on dist-test-slave-rxv6"
server_key: "76954dacb2da43d3da7a124a03d40688"
server_key_iv: "9d1857093ad7a06e24e29903822e4129"
server_key_version: "encryptionkey@0"
I20260619 18:37:39.199931 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:39.210691 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:39.213546 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:39.213716 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:39.215265 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:39.215313 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:39.215374 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:39.215399 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:37:39 dist-test-slave-rxv6 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:39.224814 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38923
I20260619 18:37:39.224907 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38923 every 8 connection(s)
I20260619 18:37:39.225731 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:37:39.228070 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.215916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48391 (local address 127.0.98.62:38207)
0619 18:37:39.216035 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:39.216040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:39.216963 (+ 923us) server_negotiation.cc:407] Connection header received
0619 18:37:39.218048 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:39.218053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:39.218111 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:39.218167 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:39.219802 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.220582 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.221757 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.222049 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.222137 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.224708 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:39.224712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:39.224714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:39.224742 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:39.226231 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:39.226768 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:39.226771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:39.226772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:39.226854 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:39.227152 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:39.227154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:39.227156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:39.227334 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:39.227448 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:39.227853 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 18:37:39.227952 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 18:37:39.228909 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.216376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38207 (local address 127.0.98.3:48391)
0619 18:37:39.216812 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:39.216860 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:39.217779 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:39.218591 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:39.218601 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:39.219005 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:39.219656 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.219676 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.220694 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.220698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.221641 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.221651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.222199 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.222203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.222368 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.222949 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:39.222969 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:39.224509 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:39.226368 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:39.226374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:39.226388 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:39.226668 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:39.226940 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:39.226942 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:39.226943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:39.227080 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:39.227445 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:39.227449 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:39.227667 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 18:37:39.227820 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 18:37:39.228062 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":99,"threads_started":1}
I20260619 18:37:39.230197 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38207
I20260619 18:37:39.230396 1990 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:39.230878 1990 heartbeater.cc:511] Master 127.0.98.62:38207 requested a full tablet report, sending...
I20260619 18:37:39.231930 1507 ts_manager.cc:194] Registered new tserver with Master: 8dc04b1c3b914c20981c740d78926d06 (127.0.98.3:38923)
I20260619 18:37:39.232650 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48391
I20260619 18:37:39.235445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 1859
I20260619 18:37:39.235543 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:37:39.235829 392 external_mini_cluster.cc:1428] Setting key 5cbf678698f069f9f050386029fe2ca2
I20260619 18:37:39.237969 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:37:39.239290 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.232176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54073 (local address 127.0.98.62:38207)
0619 18:37:39.232285 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:39.232288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:39.232485 (+ 197us) server_negotiation.cc:407] Connection header received
0619 18:37:39.232560 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:39.232563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:39.232603 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:39.232684 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:39.233086 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.233629 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.234751 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.234971 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.235050 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.236229 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:39.236232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:39.236235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:39.236317 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:39.237786 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:39.238221 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:39.238225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:39.238227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:39.238310 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:39.238542 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:39.238546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:39.238547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:39.238711 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:39.238792 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:39.239039 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 18:37:39.239137 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:37:39.239300 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.232128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38207 (local address 127.0.98.3:54073)
0619 18:37:39.232413 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:39.232425 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:39.232502 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:39.232688 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:39.232690 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:39.232879 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:39.232988 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.232994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.233762 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.233766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.234632 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.234642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.235094 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.235097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.235168 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.235669 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:39.235684 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:39.236086 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:39.237879 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:39.237882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:39.237885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:39.238131 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:39.238386 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:39.238389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:39.238391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:39.238468 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:39.238795 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:39.238798 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:39.238884 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 18:37:39.238966 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 18:37:39.239102 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
Jun 19 18:37:39 dist-test-slave-rxv6 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894259, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:37:39 dist-test-slave-rxv6 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894259, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:39.272104 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.263140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35582 (local address 127.0.98.62:38207)
0619 18:37:39.263291 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:39.263296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:39.263315 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:37:39.263372 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:39.263376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:39.263476 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:39.263577 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:39.264463 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.264996 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.266219 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.266443 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.266530 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.268979 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:39.268982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:39.268984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:39.269019 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:39.270483 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:39.270887 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:39.270889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:39.270891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:39.270964 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:39.271314 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:39.271315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:39.271317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:39.271487 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:39.271589 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:39.271848 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 18:37:39.271944 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:37:39.275221 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:37:39.278244 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:37:39.292546 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.286815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38076 (local address 127.0.98.1:35651)
0619 18:37:39.287532 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:39.287552 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:39.287570 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:37:39.287629 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:39.287632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:39.287824 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:39.287977 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:39.288288 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.289476 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.291153 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.291921 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.292030 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.292199 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:39.292333 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 18:37:39.292419 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":74,"threads_started":1}
I20260619 18:37:39.292577 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.286660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35651 (local address 127.0.0.1:38076)
0619 18:37:39.287103 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:39.287154 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:39.287281 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:39.287977 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:39.287983 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:39.288007 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:39.288152 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.288161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.289584 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.289588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.290925 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.290934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.292039 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.292042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.292148 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.292189 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 18:37:39.292308 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 18:37:39.292431 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":145,"thread_start_us":70,"threads_started":1}
I20260619 18:37:39.293018 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.286729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38923 (local address 127.0.0.1:52678)
0619 18:37:39.287323 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:39.287337 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:39.287412 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:39.289062 (+ 1650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:39.289066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:39.289081 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:39.289167 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.289173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.290396 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.290399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.291695 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.291703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.292631 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.292635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.292732 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.292754 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:37:39.292822 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 18:37:39.292893 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":330,"thread_start_us":99,"threads_started":1}
I20260619 18:37:39.293080 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.288078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52678 (local address 127.0.98.3:38923)
0619 18:37:39.288710 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:39.288722 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:39.288743 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:37:39.288816 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:39.288820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:39.288946 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:39.289069 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:39.289270 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.290280 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.291801 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.292508 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.292608 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.292765 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:39.292849 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 18:37:39.292941 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":89,"threads_started":1}
I20260619 18:37:39.293282 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.287360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34839 (local address 127.0.0.1:35882)
0619 18:37:39.287670 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:39.287683 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:39.287760 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:39.288357 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:39.288360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:39.288379 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:39.288466 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.288472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.289749 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.289752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.290780 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.290789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.292925 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.292929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.293013 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.293035 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:37:39.293102 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 18:37:39.293169 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260619 18:37:39.293586 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.287594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.0.98.2:34839)
0619 18:37:39.287889 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:39.287900 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:39.287921 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:37:39.287996 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:39.288001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:39.288182 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:39.288332 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:39.288603 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.289653 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.290897 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.291921 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.292030 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.292199 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:39.293404 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0619 18:37:39.293486 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":97,"threads_started":1}
I20260619 18:37:39.295691 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 190869cf52134be6b2813dccf1930ff3 (DEFAULT_TABLE table=test-table [id=afe952e528ca49aaab33a9c196210c1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:39.295946 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 190869cf52134be6b2813dccf1930ff3 (DEFAULT_TABLE table=test-table [id=afe952e528ca49aaab33a9c196210c1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:39.296788 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 190869cf52134be6b2813dccf1930ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:39.297039 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 190869cf52134be6b2813dccf1930ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:39.298205 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 190869cf52134be6b2813dccf1930ff3 (DEFAULT_TABLE table=test-table [id=afe952e528ca49aaab33a9c196210c1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:39.299221 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 190869cf52134be6b2813dccf1930ff3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:39.303623 2013 tablet_bootstrap.cc:492] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b: Bootstrap starting.
I20260619 18:37:39.304951 2014 tablet_bootstrap.cc:492] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9: Bootstrap starting.
I20260619 18:37:39.306169 2013 tablet_bootstrap.cc:654] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:39.306671 2015 tablet_bootstrap.cc:492] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06: Bootstrap starting.
I20260619 18:37:39.307233 2013 log.cc:824] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:39.307459 2014 tablet_bootstrap.cc:654] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:39.308168 2014 log.cc:824] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:39.308530 2015 tablet_bootstrap.cc:654] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:39.309198 2015 log.cc:824] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:39.309288 2013 tablet_bootstrap.cc:492] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b: No bootstrap required, opened a new log
I20260619 18:37:39.309479 2013 ts_tablet_manager.cc:1397] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:37:39.310094 2014 tablet_bootstrap.cc:492] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9: No bootstrap required, opened a new log
I20260619 18:37:39.310268 2014 ts_tablet_manager.cc:1397] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 18:37:39.310822 2015 tablet_bootstrap.cc:492] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06: No bootstrap required, opened a new log
I20260619 18:37:39.310981 2015 ts_tablet_manager.cc:1397] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 18:37:39.313412 2013 raft_consensus.cc:348] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } }
I20260619 18:37:39.313692 2013 raft_consensus.cc:374] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:39.313750 2013 raft_consensus.cc:729] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2fb3b4f805341499b148957933cc70b, State: Initialized, Role: FOLLOWER
I20260619 18:37:39.313961 2014 raft_consensus.cc:348] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } }
I20260619 18:37:39.314213 2014 raft_consensus.cc:374] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:39.314261 2014 raft_consensus.cc:729] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7752b495734f4010920a0ffec22d73b9, State: Initialized, Role: FOLLOWER
I20260619 18:37:39.314237 2013 consensus_queue.cc:260] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } }
I20260619 18:37:39.314394 2015 raft_consensus.cc:348] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } }
I20260619 18:37:39.314626 2015 raft_consensus.cc:374] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:39.314682 2015 raft_consensus.cc:729] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc04b1c3b914c20981c740d78926d06, State: Initialized, Role: FOLLOWER
I20260619 18:37:39.314707 2014 consensus_queue.cc:260] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } }
I20260619 18:37:39.315133 1850 heartbeater.cc:503] Master 127.0.98.62:38207 was elected leader, sending a full tablet report...
I20260619 18:37:39.315110 2015 consensus_queue.cc:260] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } }
I20260619 18:37:39.315708 2014 ts_tablet_manager.cc:1428] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 18:37:39.315933 2015 ts_tablet_manager.cc:1428] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 18:37:39.316069 1710 heartbeater.cc:503] Master 127.0.98.62:38207 was elected leader, sending a full tablet report...
I20260619 18:37:39.316164 1990 heartbeater.cc:503] Master 127.0.98.62:38207 was elected leader, sending a full tablet report...
I20260619 18:37:39.317203 2013 ts_tablet_manager.cc:1428] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260619 18:37:39.390540 2020 raft_consensus.cc:482] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:37:39.390748 2020 raft_consensus.cc:504] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } }
I20260619 18:37:39.391913 2020 leader_election.cc:288] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dc04b1c3b914c20981c740d78926d06 (127.0.98.3:38923), e2fb3b4f805341499b148957933cc70b (127.0.98.2:34839)
I20260619 18:37:39.395592 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.392155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38923 (local address 127.0.98.1:54405)
0619 18:37:39.392393 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:39.392409 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:39.392516 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:39.392898 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:39.392899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:39.392915 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:39.393013 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.393018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.393744 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.393748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.394575 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.394584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.395202 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.395205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.395292 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.395311 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 18:37:39.395374 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 18:37:39.395441 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
I20260619 18:37:39.395577 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.392285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54405 (local address 127.0.98.3:38923)
0619 18:37:39.392400 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:39.392403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:39.392470 (+ 67us) server_negotiation.cc:407] Connection header received
0619 18:37:39.392659 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:39.392663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:39.392715 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:39.392764 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:39.393121 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.393643 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.394701 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.395082 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.395173 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.395236 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:39.395402 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 18:37:39.395469 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:37:39.396205 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.392519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34839 (local address 127.0.98.1:40783)
0619 18:37:39.392778 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:39.392792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:39.392897 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:39.393154 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:39.393157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:39.393172 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:39.393261 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.393266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.394033 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.394036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.394763 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.394771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.395356 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.395360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.395923 (+ 563us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.395943 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 18:37:39.396014 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 18:37:39.396077 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":71,"threads_started":1}
I20260619 18:37:39.396242 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.392658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40783 (local address 127.0.98.2:34839)
0619 18:37:39.392778 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:39.392792 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:39.392850 (+ 58us) server_negotiation.cc:407] Connection header received
0619 18:37:39.393011 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:39.393014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:39.393060 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:39.393121 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:39.393401 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.393931 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.394879 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.395251 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.395340 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.395397 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:39.396084 (+ 687us) server_negotiation.cc:299] Negotiation successful
0619 18:37:39.396150 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:37:39.396293 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190869cf52134be6b2813dccf1930ff3" candidate_uuid: "7752b495734f4010920a0ffec22d73b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc04b1c3b914c20981c740d78926d06" is_pre_election: true
I20260619 18:37:39.396623 1944 raft_consensus.cc:2454] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7752b495734f4010920a0ffec22d73b9 in term 0.
I20260619 18:37:39.396889 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190869cf52134be6b2813dccf1930ff3" candidate_uuid: "7752b495734f4010920a0ffec22d73b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2fb3b4f805341499b148957933cc70b" is_pre_election: true
I20260619 18:37:39.397110 1597 leader_election.cc:302] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7752b495734f4010920a0ffec22d73b9, 8dc04b1c3b914c20981c740d78926d06; no voters:
I20260619 18:37:39.397171 1804 raft_consensus.cc:2454] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7752b495734f4010920a0ffec22d73b9 in term 0.
I20260619 18:37:39.397400 2020 raft_consensus.cc:2781] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:37:39.397480 2020 raft_consensus.cc:482] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:37:39.397528 2020 raft_consensus.cc:3037] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:39.398581 2020 raft_consensus.cc:504] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } }
I20260619 18:37:39.398903 2020 leader_election.cc:288] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [CANDIDATE]: Term 1 election: Requested vote from peers 8dc04b1c3b914c20981c740d78926d06 (127.0.98.3:38923), e2fb3b4f805341499b148957933cc70b (127.0.98.2:34839)
I20260619 18:37:39.399224 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190869cf52134be6b2813dccf1930ff3" candidate_uuid: "7752b495734f4010920a0ffec22d73b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc04b1c3b914c20981c740d78926d06"
I20260619 18:37:39.399297 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190869cf52134be6b2813dccf1930ff3" candidate_uuid: "7752b495734f4010920a0ffec22d73b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2fb3b4f805341499b148957933cc70b"
I20260619 18:37:39.399341 1944 raft_consensus.cc:3037] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:39.399431 1804 raft_consensus.cc:3037] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:39.400463 1804 raft_consensus.cc:2454] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7752b495734f4010920a0ffec22d73b9 in term 1.
I20260619 18:37:39.400645 1944 raft_consensus.cc:2454] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7752b495734f4010920a0ffec22d73b9 in term 1.
I20260619 18:37:39.400750 1597 leader_election.cc:302] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7752b495734f4010920a0ffec22d73b9, e2fb3b4f805341499b148957933cc70b; no voters:
I20260619 18:37:39.400913 2020 raft_consensus.cc:2781] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:39.401153 2020 raft_consensus.cc:686] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [term 1 LEADER]: Becoming Leader. State: Replica: 7752b495734f4010920a0ffec22d73b9, State: Running, Role: LEADER
I20260619 18:37:39.401528 2020 consensus_queue.cc:237] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } }
I20260619 18:37:39.404820 1506 catalog_manager.cc:5495] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7752b495734f4010920a0ffec22d73b9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7752b495734f4010920a0ffec22d73b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } health_report { overall_health: UNKNOWN } } }
I20260619 18:37:39.423386 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.419678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.0.98.1:35651)
0619 18:37:39.419804 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:39.419808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:39.419822 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:37:39.420019 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:39.420021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:39.420078 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:39.420161 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:37:39.420420 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.420918 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.421858 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.422115 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.422199 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.422547 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:37:39.422990 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:37:39.423094 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:39.423184 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 18:37:39.423242 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:37:39.428941 1804 raft_consensus.cc:1261] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b [term 1 FOLLOWER]: Refusing update from remote peer 7752b495734f4010920a0ffec22d73b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:39.428982 1944 raft_consensus.cc:1261] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06 [term 1 FOLLOWER]: Refusing update from remote peer 7752b495734f4010920a0ffec22d73b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:39.429587 2020 consensus_queue.cc:1035] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:39.429788 2024 consensus_queue.cc:1035] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:39.437625 2028 mvcc.cc:205] Tried to move back new op lower bound from 7298638886614515712 to 7298638886511681536. Current Snapshot: MvccSnapshot[applied={T|T < 7298638886614515712}]
I20260619 18:37:39.438087 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298638886614515712 to 7298638886511681536. Current Snapshot: MvccSnapshot[applied={T|T < 7298638886614515712}]
I20260619 18:37:39.449661 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:37:39.449837 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:37:39.452199 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063: Sending DeleteTablet for 3 replicas of tablet 190869cf52134be6b2813dccf1930ff3
I20260619 18:37:39.453032 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 190869cf52134be6b2813dccf1930ff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52678
I20260619 18:37:39.453070 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 190869cf52134be6b2813dccf1930ff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38076
I20260619 18:37:39.453171 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 190869cf52134be6b2813dccf1930ff3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35882
I20260619 18:37:39.453629 2037 tablet_replica.cc:330] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06: stopping tablet replica
I20260619 18:37:39.453959 2037 raft_consensus.cc:2229] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:37:39.454299 2037 raft_consensus.cc:2258] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:39.454423 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 18:37:39.454668 2038 tablet_replica.cc:330] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9: stopping tablet replica
W20260619 18:37:39.454833 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:37:39.454946 2038 raft_consensus.cc:2229] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:37:39.455345 2038 raft_consensus.cc:2258] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:39.456182 2039 tablet_replica.cc:330] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b: stopping tablet replica
I20260619 18:37:39.456408 2037 ts_tablet_manager.cc:1905] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:39.456456 2039 raft_consensus.cc:2229] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:37:39.456740 2039 raft_consensus.cc:2258] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:39.458499 2038 ts_tablet_manager.cc:1905] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:39.459275 2037 ts_tablet_manager.cc:1918] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:37:39.459412 2037 log.cc:1196] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/190869cf52134be6b2813dccf1930ff3
I20260619 18:37:39.459798 2037 ts_tablet_manager.cc:1939] T 190869cf52134be6b2813dccf1930ff3 P 8dc04b1c3b914c20981c740d78926d06: Deleting consensus metadata
I20260619 18:37:39.460495 1784 tablet_service.cc:1467] Processing CreateTablet for tablet e362b5e6b6ba46948d6467df0cab68cf (DEFAULT_TABLE table=test-table [id=a17b6fa243c846b2a0d4562165a4ddda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:39.460767 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e362b5e6b6ba46948d6467df0cab68cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:39.461088 2038 ts_tablet_manager.cc:1918] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:37:39.461176 2038 log.cc:1196] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/wal/wals/190869cf52134be6b2813dccf1930ff3
I20260619 18:37:39.461380 1493 catalog_manager.cc:4841] TS 8dc04b1c3b914c20981c740d78926d06 (127.0.98.3:38923): tablet 190869cf52134be6b2813dccf1930ff3 (table test-table [id=afe952e528ca49aaab33a9c196210c1c]) successfully deleted
I20260619 18:37:39.461474 2038 ts_tablet_manager.cc:1939] T 190869cf52134be6b2813dccf1930ff3 P 7752b495734f4010920a0ffec22d73b9: Deleting consensus metadata
I20260619 18:37:39.462393 1924 tablet_service.cc:1467] Processing CreateTablet for tablet e362b5e6b6ba46948d6467df0cab68cf (DEFAULT_TABLE table=test-table [id=a17b6fa243c846b2a0d4562165a4ddda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:39.462641 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e362b5e6b6ba46948d6467df0cab68cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:39.463166 2013 tablet_bootstrap.cc:492] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b: Bootstrap starting.
I20260619 18:37:39.464004 2039 ts_tablet_manager.cc:1905] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:39.464115 2013 tablet_bootstrap.cc:654] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:39.464964 2015 tablet_bootstrap.cc:492] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06: Bootstrap starting.
I20260619 18:37:39.465371 1644 tablet_service.cc:1467] Processing CreateTablet for tablet e362b5e6b6ba46948d6467df0cab68cf (DEFAULT_TABLE table=test-table [id=a17b6fa243c846b2a0d4562165a4ddda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:39.465605 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e362b5e6b6ba46948d6467df0cab68cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:39.465858 2015 tablet_bootstrap.cc:654] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:39.466341 2039 ts_tablet_manager.cc:1918] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:37:39.466421 2039 log.cc:1196] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/190869cf52134be6b2813dccf1930ff3
I20260619 18:37:39.466688 2039 ts_tablet_manager.cc:1939] T 190869cf52134be6b2813dccf1930ff3 P e2fb3b4f805341499b148957933cc70b: Deleting consensus metadata
I20260619 18:37:39.466679 1491 catalog_manager.cc:4841] TS 7752b495734f4010920a0ffec22d73b9 (127.0.98.1:35651): tablet 190869cf52134be6b2813dccf1930ff3 (table test-table [id=afe952e528ca49aaab33a9c196210c1c]) successfully deleted
I20260619 18:37:39.467695 2015 tablet_bootstrap.cc:492] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06: No bootstrap required, opened a new log
I20260619 18:37:39.467741 2014 tablet_bootstrap.cc:492] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9: Bootstrap starting.
I20260619 18:37:39.467772 2015 ts_tablet_manager.cc:1397] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:37:39.467993 1493 catalog_manager.cc:4841] TS e2fb3b4f805341499b148957933cc70b (127.0.98.2:34839): tablet 190869cf52134be6b2813dccf1930ff3 (table test-table [id=afe952e528ca49aaab33a9c196210c1c]) successfully deleted
I20260619 18:37:39.468200 2015 raft_consensus.cc:348] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:39.468307 2015 raft_consensus.cc:374] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:39.468335 2015 raft_consensus.cc:729] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc04b1c3b914c20981c740d78926d06, State: Initialized, Role: FOLLOWER
I20260619 18:37:39.468453 2015 consensus_queue.cc:260] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:39.468647 2014 tablet_bootstrap.cc:654] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:39.468699 2015 ts_tablet_manager.cc:1428] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:37:39.469414 2013 tablet_bootstrap.cc:492] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b: No bootstrap required, opened a new log
I20260619 18:37:39.469509 2013 ts_tablet_manager.cc:1397] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 18:37:39.469756 2014 tablet_bootstrap.cc:492] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9: No bootstrap required, opened a new log
I20260619 18:37:39.469851 2014 ts_tablet_manager.cc:1397] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:37:39.469992 2013 raft_consensus.cc:348] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:39.470117 2013 raft_consensus.cc:374] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:39.470158 2013 raft_consensus.cc:729] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2fb3b4f805341499b148957933cc70b, State: Initialized, Role: FOLLOWER
I20260619 18:37:39.470332 2014 raft_consensus.cc:348] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:39.470350 2013 consensus_queue.cc:260] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:39.470438 2014 raft_consensus.cc:374] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:39.470474 2014 raft_consensus.cc:729] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7752b495734f4010920a0ffec22d73b9, State: Initialized, Role: FOLLOWER
I20260619 18:37:39.470577 2014 consensus_queue.cc:260] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:39.470825 2014 ts_tablet_manager.cc:1428] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:37:39.471738 2013 ts_tablet_manager.cc:1428] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260619 18:37:39.476706 1991 tablet.cc:2369] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:37:39.518851 1851 tablet.cc:2369] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:37:39.554584 1711 tablet.cc:2369] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:37:39.713555 2021 raft_consensus.cc:482] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:37:39.713697 2021 raft_consensus.cc:504] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:39.714753 2021 leader_election.cc:288] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7752b495734f4010920a0ffec22d73b9 (127.0.98.1:35651), e2fb3b4f805341499b148957933cc70b (127.0.98.2:34839)
I20260619 18:37:39.719587 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.715106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34907 (local address 127.0.98.2:34839)
0619 18:37:39.715211 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:39.715215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:39.715672 (+ 457us) server_negotiation.cc:407] Connection header received
0619 18:37:39.715769 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:39.715772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:39.715811 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:39.715903 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:39.716133 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.716815 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.718279 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.718908 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.719034 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.719109 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:39.719384 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 18:37:39.719454 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:37:39.719609 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.715022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34839 (local address 127.0.98.3:34907)
0619 18:37:39.715567 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:39.715581 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:39.715684 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:39.715919 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:39.715922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:39.715940 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:39.716037 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.716042 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.716930 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.716933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.718162 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.718168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.719072 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.719076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.719246 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.719293 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 18:37:39.719366 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 18:37:39.719439 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":292,"thread_start_us":94,"threads_started":1}
I20260619 18:37:39.719731 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.715173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49121 (local address 127.0.98.1:35651)
0619 18:37:39.715301 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:39.715305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:39.715325 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:37:39.715462 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:39.715465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:39.715517 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:39.715584 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:39.715877 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.716621 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.718279 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.718966 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.719058 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.719113 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:39.719444 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 18:37:39.719628 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:37:39.720093 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.715022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35651 (local address 127.0.98.3:49121)
0619 18:37:39.715176 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:39.715191 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:39.715290 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:39.715617 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:39.715620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:39.715641 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:39.715776 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.715783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.716749 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.716752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.718132 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.718143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.719072 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.719076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.719249 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.719288 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 18:37:39.719421 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 18:37:39.719477 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":46,"spinlock_wait_cycles":11648}
I20260619 18:37:39.720173 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e362b5e6b6ba46948d6467df0cab68cf" candidate_uuid: "8dc04b1c3b914c20981c740d78926d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2fb3b4f805341499b148957933cc70b" is_pre_election: true
I20260619 18:37:39.720330 1804 raft_consensus.cc:2454] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dc04b1c3b914c20981c740d78926d06 in term 0.
I20260619 18:37:39.720587 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e362b5e6b6ba46948d6467df0cab68cf" candidate_uuid: "8dc04b1c3b914c20981c740d78926d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7752b495734f4010920a0ffec22d73b9" is_pre_election: true
I20260619 18:37:39.720686 1877 leader_election.cc:302] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dc04b1c3b914c20981c740d78926d06, e2fb3b4f805341499b148957933cc70b; no voters:
I20260619 18:37:39.720837 1664 raft_consensus.cc:2454] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dc04b1c3b914c20981c740d78926d06 in term 0.
I20260619 18:37:39.720993 2021 raft_consensus.cc:2781] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:37:39.721069 2021 raft_consensus.cc:482] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:37:39.721120 2021 raft_consensus.cc:3037] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:39.722221 2021 raft_consensus.cc:504] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:39.722688 2021 leader_election.cc:288] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [CANDIDATE]: Term 1 election: Requested vote from peers 7752b495734f4010920a0ffec22d73b9 (127.0.98.1:35651), e2fb3b4f805341499b148957933cc70b (127.0.98.2:34839)
I20260619 18:37:39.723076 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e362b5e6b6ba46948d6467df0cab68cf" candidate_uuid: "8dc04b1c3b914c20981c740d78926d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2fb3b4f805341499b148957933cc70b"
I20260619 18:37:39.723194 1804 raft_consensus.cc:3037] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:39.723168 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e362b5e6b6ba46948d6467df0cab68cf" candidate_uuid: "8dc04b1c3b914c20981c740d78926d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7752b495734f4010920a0ffec22d73b9"
I20260619 18:37:39.723275 1664 raft_consensus.cc:3037] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:39.724165 1804 raft_consensus.cc:2454] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dc04b1c3b914c20981c740d78926d06 in term 1.
I20260619 18:37:39.724231 1664 raft_consensus.cc:2454] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dc04b1c3b914c20981c740d78926d06 in term 1.
I20260619 18:37:39.724419 1877 leader_election.cc:302] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dc04b1c3b914c20981c740d78926d06, e2fb3b4f805341499b148957933cc70b; no voters:
I20260619 18:37:39.724579 2021 raft_consensus.cc:2781] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:39.724746 2021 raft_consensus.cc:686] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [term 1 LEADER]: Becoming Leader. State: Replica: 8dc04b1c3b914c20981c740d78926d06, State: Running, Role: LEADER
I20260619 18:37:39.724980 2021 consensus_queue.cc:237] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:39.727176 1506 catalog_manager.cc:5495] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8dc04b1c3b914c20981c740d78926d06 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8dc04b1c3b914c20981c740d78926d06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } health_report { overall_health: HEALTHY } } }
Jun 19 18:37:39 dist-test-slave-rxv6 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:39.796756 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.787613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38207 (local address 127.0.0.1:35598)
0619 18:37:39.787901 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:39.787914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:39.788018 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:39.788270 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:39.788273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:39.788679 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:39.788781 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.788787 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.789611 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.789614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.790236 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:39.790245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.790736 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.790739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:39.790805 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.791389 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:39.791401 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:39.792949 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:39.795002 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:39.795013 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:39.795022 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:39.795323 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:39.795597 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:39.795600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:39.795602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:39.795677 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:39.796134 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:39.796142 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:39.796265 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 18:37:39.796379 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 18:37:39.796539 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":110,"threads_started":1}
I20260619 18:37:39.797405 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:39.787678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.0.98.62:38207)
0619 18:37:39.787931 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:39.787938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:39.787970 (+ 32us) server_negotiation.cc:407] Connection header received
0619 18:37:39.788095 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:39.788097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:39.788137 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:39.788223 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:39.788902 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.789504 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.790386 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:39.790608 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:39.790699 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:39.793105 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:39.793109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:39.793112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:39.793140 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:39.794876 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:39.795427 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:39.795430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:39.795432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:39.795512 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:39.795756 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:39.795759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:39.795761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:39.795995 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:39.796087 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:39.797125 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0619 18:37:39.797247 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":89,"threads_started":1}
I20260619 18:37:39.805574 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35598:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 18:37:39.806376 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:37:39.810833 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 7d4b28a77094422cabe837507950292c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=381c74f10b3d43248f40f461795e2233]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:37:39.811017 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 7d4b28a77094422cabe837507950292c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=381c74f10b3d43248f40f461795e2233]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:37:39.811177 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d4b28a77094422cabe837507950292c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:39.811336 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d4b28a77094422cabe837507950292c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:39.813380 2014 tablet_bootstrap.cc:492] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9: Bootstrap starting.
I20260619 18:37:39.813850 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 7d4b28a77094422cabe837507950292c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=381c74f10b3d43248f40f461795e2233]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:37:39.814225 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d4b28a77094422cabe837507950292c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:39.814227 2014 tablet_bootstrap.cc:654] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:39.814414 2013 tablet_bootstrap.cc:492] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b: Bootstrap starting.
I20260619 18:37:39.815292 2013 tablet_bootstrap.cc:654] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:39.816504 2015 tablet_bootstrap.cc:492] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06: Bootstrap starting.
I20260619 18:37:39.817384 2015 tablet_bootstrap.cc:654] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:39.817404 2014 tablet_bootstrap.cc:492] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9: No bootstrap required, opened a new log
I20260619 18:37:39.817476 2014 ts_tablet_manager.cc:1397] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 18:37:39.817898 2014 raft_consensus.cc:348] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:39.818073 2014 raft_consensus.cc:374] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:39.818110 2014 raft_consensus.cc:729] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7752b495734f4010920a0ffec22d73b9, State: Initialized, Role: FOLLOWER
I20260619 18:37:39.818243 2014 consensus_queue.cc:260] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:39.818512 2014 ts_tablet_manager.cc:1428] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:37:39.818727 2015 tablet_bootstrap.cc:492] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06: No bootstrap required, opened a new log
I20260619 18:37:39.818699 2020 tablet_replica.cc:439] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } }
I20260619 18:37:39.818799 2015 ts_tablet_manager.cc:1397] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:37:39.818826 2020 tablet_replica.cc:442] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:37:39.818830 2013 tablet_bootstrap.cc:492] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b: No bootstrap required, opened a new log
I20260619 18:37:39.818902 2013 ts_tablet_manager.cc:1397] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 18:37:39.819212 2015 raft_consensus.cc:348] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:39.819314 2015 raft_consensus.cc:374] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:39.819339 2015 raft_consensus.cc:729] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc04b1c3b914c20981c740d78926d06, State: Initialized, Role: FOLLOWER
I20260619 18:37:39.819305 2013 raft_consensus.cc:348] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:39.819445 2015 consensus_queue.cc:260] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:39.819495 2013 raft_consensus.cc:374] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:39.819523 2013 raft_consensus.cc:729] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2fb3b4f805341499b148957933cc70b, State: Initialized, Role: FOLLOWER
I20260619 18:37:39.819662 2013 consensus_queue.cc:260] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:39.819669 2021 tablet_replica.cc:439] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } }
I20260619 18:37:39.819793 2021 tablet_replica.cc:442] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:37:39.819936 2013 ts_tablet_manager.cc:1428] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:37:39.820526 2019 tablet_replica.cc:439] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } }
I20260619 18:37:39.820648 2019 tablet_replica.cc:442] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:37:39.819676 2015 ts_tablet_manager.cc:1428] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:37:40.034308 2019 raft_consensus.cc:482] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:37:40.034457 2019 raft_consensus.cc:504] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:40.035581 2019 leader_election.cc:288] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7752b495734f4010920a0ffec22d73b9 (127.0.98.1:35651), 8dc04b1c3b914c20981c740d78926d06 (127.0.98.3:38923)
I20260619 18:37:40.039326 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:40.036162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41147 (local address 127.0.98.1:35651)
0619 18:37:40.036274 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:40.036279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:40.036292 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:37:40.036340 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:40.036342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:40.036382 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:40.036444 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:40.036726 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.037228 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.038390 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.038786 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.038876 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:40.038937 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:40.039141 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 18:37:40.039204 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:37:40.039330 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:40.035917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47203 (local address 127.0.98.3:38923)
0619 18:37:40.036175 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:40.036179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:40.036277 (+ 98us) server_negotiation.cc:407] Connection header received
0619 18:37:40.036402 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:40.036405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:40.036449 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:40.036531 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:40.036785 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.037298 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.038393 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.038796 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.038885 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:40.038945 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:40.039120 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 18:37:40.039190 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":93,"threads_started":1}
I20260619 18:37:40.039338 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:40.035805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35651 (local address 127.0.98.2:41147)
0619 18:37:40.036050 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:40.036069 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:40.036192 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:40.036489 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:40.036492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:40.036508 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:40.036626 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:40.036631 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.037332 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.037335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:40.038276 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:40.038285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.038894 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.038898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:40.039036 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:40.039052 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 18:37:40.039115 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 18:37:40.039201 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":51,"thread_start_us":74,"threads_started":1}
I20260619 18:37:40.039340 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:40.035845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38923 (local address 127.0.98.2:47203)
0619 18:37:40.036206 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:40.036219 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:40.036315 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:40.036547 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:40.036550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:40.036570 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:40.036684 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:40.036690 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.037407 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.037411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:40.038268 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:40.038278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.038899 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.038902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:40.039011 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:40.039033 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:37:40.039098 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 18:37:40.039191 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":43,"thread_start_us":76,"threads_started":1}
I20260619 18:37:40.039840 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d4b28a77094422cabe837507950292c" candidate_uuid: "e2fb3b4f805341499b148957933cc70b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7752b495734f4010920a0ffec22d73b9" is_pre_election: true
I20260619 18:37:40.039974 1664 raft_consensus.cc:2454] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2fb3b4f805341499b148957933cc70b in term 0.
I20260619 18:37:40.040132 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d4b28a77094422cabe837507950292c" candidate_uuid: "e2fb3b4f805341499b148957933cc70b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc04b1c3b914c20981c740d78926d06" is_pre_election: true
I20260619 18:37:40.040266 1944 raft_consensus.cc:2454] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2fb3b4f805341499b148957933cc70b in term 0.
I20260619 18:37:40.040315 1735 leader_election.cc:302] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7752b495734f4010920a0ffec22d73b9, e2fb3b4f805341499b148957933cc70b; no voters:
I20260619 18:37:40.040560 2019 raft_consensus.cc:2781] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:37:40.040616 2019 raft_consensus.cc:482] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:37:40.040663 2019 raft_consensus.cc:3037] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:40.041597 2019 raft_consensus.cc:504] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:40.041967 2019 leader_election.cc:288] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [CANDIDATE]: Term 1 election: Requested vote from peers 7752b495734f4010920a0ffec22d73b9 (127.0.98.1:35651), 8dc04b1c3b914c20981c740d78926d06 (127.0.98.3:38923)
I20260619 18:37:40.042330 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d4b28a77094422cabe837507950292c" candidate_uuid: "e2fb3b4f805341499b148957933cc70b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc04b1c3b914c20981c740d78926d06"
I20260619 18:37:40.042349 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d4b28a77094422cabe837507950292c" candidate_uuid: "e2fb3b4f805341499b148957933cc70b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7752b495734f4010920a0ffec22d73b9"
I20260619 18:37:40.042440 1944 raft_consensus.cc:3037] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:40.042447 1664 raft_consensus.cc:3037] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:40.043311 1944 raft_consensus.cc:2454] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2fb3b4f805341499b148957933cc70b in term 1.
I20260619 18:37:40.043576 1737 leader_election.cc:302] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dc04b1c3b914c20981c740d78926d06, e2fb3b4f805341499b148957933cc70b; no voters:
I20260619 18:37:40.043725 2019 raft_consensus.cc:2781] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:40.043812 1664 raft_consensus.cc:2454] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2fb3b4f805341499b148957933cc70b in term 1.
I20260619 18:37:40.043881 2019 raft_consensus.cc:686] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [term 1 LEADER]: Becoming Leader. State: Replica: e2fb3b4f805341499b148957933cc70b, State: Running, Role: LEADER
I20260619 18:37:40.044126 2019 consensus_queue.cc:237] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } }
I20260619 18:37:40.045293 2059 tablet_replica.cc:439] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b: TxnStatusTablet state changed. Reason: New leader e2fb3b4f805341499b148957933cc70b. Latest consensus state: current_term: 1 leader_uuid: "e2fb3b4f805341499b148957933cc70b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } }
I20260619 18:37:40.045449 2059 tablet_replica.cc:442] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:37:40.045759 2061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 18:37:40.045897 2061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 18:37:40.046592 1505 catalog_manager.cc:5495] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b reported cstate change: term changed from 0 to 1, leader changed from <none> to e2fb3b4f805341499b148957933cc70b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e2fb3b4f805341499b148957933cc70b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } health_report { overall_health: UNKNOWN } } }
I20260619 18:37:40.060353 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:40.055972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34839 (local address 127.0.0.1:35892)
0619 18:37:40.056217 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:40.056232 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:40.056319 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:40.056695 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:40.056698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:40.056705 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:37:40.056800 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:40.056805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.057841 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.057844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:40.058580 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:40.058590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.059154 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.059157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:40.059230 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:40.059276 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:37:40.060041 (+ 765us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:37:40.060056 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 18:37:40.060111 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 18:37:40.060200 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260619 18:37:40.060343 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:40.056123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.0.98.2:34839)
0619 18:37:40.056249 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:40.056254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:40.056270 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:37:40.056449 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:40.056452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:40.056545 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:40.056659 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:37:40.056923 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.057717 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.058703 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.059030 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.059135 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:40.059405 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:37:40.059935 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:37:40.060053 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:40.060139 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 18:37:40.060197 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:37:40.063642 1944 raft_consensus.cc:1261] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06 [term 1 FOLLOWER]: Refusing update from remote peer e2fb3b4f805341499b148957933cc70b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:40.063760 1664 raft_consensus.cc:1261] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9 [term 1 FOLLOWER]: Refusing update from remote peer e2fb3b4f805341499b148957933cc70b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:40.064185 2059 consensus_queue.cc:1035] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:40.064394 2019 consensus_queue.cc:1035] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:40.065791 2021 tablet_replica.cc:439] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06: TxnStatusTablet state changed. Reason: New leader e2fb3b4f805341499b148957933cc70b. Latest consensus state: current_term: 1 leader_uuid: "e2fb3b4f805341499b148957933cc70b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } }
I20260619 18:37:40.065936 2021 tablet_replica.cc:442] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:37:40.066052 2020 tablet_replica.cc:439] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9: TxnStatusTablet state changed. Reason: New leader e2fb3b4f805341499b148957933cc70b. Latest consensus state: current_term: 1 leader_uuid: "e2fb3b4f805341499b148957933cc70b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } }
I20260619 18:37:40.066157 2020 tablet_replica.cc:442] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:37:40.068203 2019 tablet_replica.cc:439] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2fb3b4f805341499b148957933cc70b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } }
I20260619 18:37:40.068310 2019 tablet_replica.cc:442] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:37:40.068650 2020 tablet_replica.cc:439] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2fb3b4f805341499b148957933cc70b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } }
I20260619 18:37:40.068746 2020 tablet_replica.cc:442] T 7d4b28a77094422cabe837507950292c P 7752b495734f4010920a0ffec22d73b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:37:40.069097 2019 tablet_replica.cc:439] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e2fb3b4f805341499b148957933cc70b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } }
I20260619 18:37:40.069193 2019 tablet_replica.cc:442] T 7d4b28a77094422cabe837507950292c P e2fb3b4f805341499b148957933cc70b: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:37:40.069548 2066 mvcc.cc:205] Tried to move back new op lower bound from 7298638889214877696 to 7298638889143087104. Current Snapshot: MvccSnapshot[applied={T|T < 7298638889214877696}]
I20260619 18:37:40.070051 2021 tablet_replica.cc:439] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e2fb3b4f805341499b148957933cc70b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 } } peers { permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 } } peers { permanent_uuid: "8dc04b1c3b914c20981c740d78926d06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38923 } } }
I20260619 18:37:40.070165 2021 tablet_replica.cc:442] T 7d4b28a77094422cabe837507950292c P 8dc04b1c3b914c20981c740d78926d06: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:37:40.082540 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:40.078677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.0.98.3:38923)
0619 18:37:40.078796 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:40.078801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:40.078911 (+ 110us) server_negotiation.cc:407] Connection header received
0619 18:37:40.079067 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:40.079070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:40.079124 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:40.079209 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:37:40.079434 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.079939 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.081087 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.081288 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.081367 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:40.081678 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:37:40.082170 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:37:40.082265 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:40.082339 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 18:37:40.082388 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 18:37:40.093518 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:40.090050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52521 (local address 127.0.98.2:34839)
0619 18:37:40.090185 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:40.090189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:40.090339 (+ 150us) server_negotiation.cc:407] Connection header received
0619 18:37:40.090450 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:40.090453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:40.090513 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:40.090593 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:37:40.090790 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.091567 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.092417 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.092748 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.092846 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:40.093108 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:37:40.093189 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:37:40.093255 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:40.093319 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 18:37:40.093367 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:37:40.093533 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:40.089946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34839 (local address 127.0.98.3:52521)
0619 18:37:40.090268 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:40.090281 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:40.090368 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:40.090603 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:40.090605 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:40.090612 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:37:40.090710 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:40.090715 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.091691 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.091694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:40.092313 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:40.092322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.092862 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.092866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:40.092937 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:40.092991 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:37:40.093260 (+ 269us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:37:40.093264 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 18:37:40.093306 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 18:37:40.093370 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":97,"threads_started":1}
I20260619 18:37:40.100255 1664 raft_consensus.cc:1261] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9 [term 1 FOLLOWER]: Refusing update from remote peer 8dc04b1c3b914c20981c740d78926d06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:40.100389 1804 raft_consensus.cc:1261] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b [term 1 FOLLOWER]: Refusing update from remote peer 8dc04b1c3b914c20981c740d78926d06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:40.100744 2044 consensus_queue.cc:1035] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7752b495734f4010920a0ffec22d73b9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:40.100986 2021 consensus_queue.cc:1035] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2fb3b4f805341499b148957933cc70b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:40.123330 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:40.119475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58819 (local address 127.0.98.3:38923)
0619 18:37:40.119601 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:40.119605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:40.119933 (+ 328us) server_negotiation.cc:407] Connection header received
0619 18:37:40.120065 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:40.120067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:40.120108 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:40.120172 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:37:40.120424 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.120976 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.122239 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.122464 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.122540 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:40.122888 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:37:40.122975 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:37:40.123018 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:40.123179 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 18:37:40.123225 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:37:40.123394 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:40.119379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38923 (local address 127.0.98.2:58819)
0619 18:37:40.119852 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:40.119868 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:40.119984 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:40.120191 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:40.120194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:40.120201 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:37:40.120319 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:40.120324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.121069 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.121072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:40.122124 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:40.122136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.122569 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.122573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:40.122683 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:40.122738 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:37:40.123076 (+ 338us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:37:40.123082 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:37:40.123149 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 18:37:40.123227 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":168,"threads_started":1}
I20260619 18:37:40.147616 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:37:40.147809 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:37:40.150182 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f499e5cf4e084db891ab671c97a8d063: Sending DeleteTablet for 3 replicas of tablet e362b5e6b6ba46948d6467df0cab68cf
I20260619 18:37:40.150779 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet e362b5e6b6ba46948d6467df0cab68cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38076
I20260619 18:37:40.150964 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet e362b5e6b6ba46948d6467df0cab68cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52678
I20260619 18:37:40.151261 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet e362b5e6b6ba46948d6467df0cab68cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35882
I20260619 18:37:40.151312 2086 tablet_replica.cc:330] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06: stopping tablet replica
I20260619 18:37:40.151430 2086 raft_consensus.cc:2229] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:37:40.151721 2086 raft_consensus.cc:2258] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:40.152434 2086 ts_tablet_manager.cc:1905] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:40.153019 2085 tablet_replica.cc:330] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9: stopping tablet replica
I20260619 18:37:40.153199 2085 raft_consensus.cc:2229] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:37:40.153368 2085 raft_consensus.cc:2258] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:40.154193 2085 ts_tablet_manager.cc:1905] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:40.155663 2086 ts_tablet_manager.cc:1918] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:37:40.155750 2086 log.cc:1196] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/e362b5e6b6ba46948d6467df0cab68cf
I20260619 18:37:40.156001 2086 ts_tablet_manager.cc:1939] T e362b5e6b6ba46948d6467df0cab68cf P 8dc04b1c3b914c20981c740d78926d06: Deleting consensus metadata
I20260619 18:37:40.156706 1493 catalog_manager.cc:4841] TS 8dc04b1c3b914c20981c740d78926d06 (127.0.98.3:38923): tablet e362b5e6b6ba46948d6467df0cab68cf (table test-table [id=a17b6fa243c846b2a0d4562165a4ddda]) successfully deleted
I20260619 18:37:40.157025 2087 tablet_replica.cc:330] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b: stopping tablet replica
I20260619 18:37:40.157161 2087 raft_consensus.cc:2229] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:37:40.157286 2087 raft_consensus.cc:2258] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:40.158120 2085 ts_tablet_manager.cc:1918] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:37:40.158250 2085 log.cc:1196] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-0/wal/wals/e362b5e6b6ba46948d6467df0cab68cf
I20260619 18:37:40.158401 2087 ts_tablet_manager.cc:1905] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:40.158607 2085 ts_tablet_manager.cc:1939] T e362b5e6b6ba46948d6467df0cab68cf P 7752b495734f4010920a0ffec22d73b9: Deleting consensus metadata
I20260619 18:37:40.159366 1491 catalog_manager.cc:4841] TS 7752b495734f4010920a0ffec22d73b9 (127.0.98.1:35651): tablet e362b5e6b6ba46948d6467df0cab68cf (table test-table [id=a17b6fa243c846b2a0d4562165a4ddda]) successfully deleted
I20260619 18:37:40.161324 2087 ts_tablet_manager.cc:1918] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:37:40.161429 2087 log.cc:1196] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/e362b5e6b6ba46948d6467df0cab68cf
I20260619 18:37:40.161697 2087 ts_tablet_manager.cc:1939] T e362b5e6b6ba46948d6467df0cab68cf P e2fb3b4f805341499b148957933cc70b: Deleting consensus metadata
I20260619 18:37:40.162546 1493 catalog_manager.cc:4841] TS e2fb3b4f805341499b148957933cc70b (127.0.98.2:34839): tablet e362b5e6b6ba46948d6467df0cab68cf (table test-table [id=a17b6fa243c846b2a0d4562165a4ddda]) successfully deleted
Jun 19 18:37:40 dist-test-slave-rxv6 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894259, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 18:37:40.183029 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:40.157807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.0.98.1:35651)
0619 18:37:40.157945 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:40.157950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:40.166257 (+ 8307us) server_negotiation.cc:407] Connection header received
0619 18:37:40.166389 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:40.166393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:40.166451 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:40.166574 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:40.167392 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.168170 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.169363 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.169729 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.169838 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:40.178603 (+ 8765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:40.178619 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:40.178637 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:40.178677 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:40.180962 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:40.181519 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:40.181526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:40.181528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:40.181625 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:40.181966 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:40.181970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:40.181972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:40.182293 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:40.182519 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:40.182730 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 18:37:40.182855 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 18:37:40.183828 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38094
I20260619 18:37:40.194958 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:40.185682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35604 (local address 127.0.98.62:38207)
0619 18:37:40.185821 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:40.185826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:40.186235 (+ 409us) server_negotiation.cc:407] Connection header received
0619 18:37:40.186416 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:40.186420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:40.186489 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:40.186573 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:40.187325 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.188161 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.189288 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:40.189661 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:40.189773 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:40.191357 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:40.191361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:40.191364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:40.191397 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:40.193171 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:40.193669 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:40.193673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:40.193675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:40.193768 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:40.194078 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:40.194081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:40.194083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:40.194294 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:40.194419 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:40.194654 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 18:37:40.194787 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 18:37:40.195449 1505 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35604
I20260619 18:37:40.197149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 1579
I20260619 18:37:40.207054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 1719
I20260619 18:37:40.213536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 1859
I20260619 18:37:40.218495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 1475
2026-06-19T18:37:40Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2934 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:37:40 dist-test-slave-rxv6 krb5kdc[2094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:37:40 dist-test-slave-rxv6 krb5kdc[2094](info): set up 2 sockets
Jun 19 18:37:40 dist-test-slave-rxv6 krb5kdc[2094](info): commencing operation
krb5kdc: starting...
W20260619 18:37:41.177675 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:37:41 dist-test-slave-rxv6 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:37:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:37:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:37:41.299059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41049
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:41.407233 2107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:41.407532 2107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:41.407593 2107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:41.411689 2107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:37:41.411768 2107 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 18:37:41.411793 2107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:41.411816 2107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:41.411837 2107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:37:41.411862 2107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:37:41.417328 2107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41049
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:41.418500 2107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:41.419579 2107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:41.428268 2115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:41.428624 2107 server_base.cc:1034] running on GCE node
W20260619 18:37:41.428272 2113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:41.428268 2112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:41.429298 2107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:41.430469 2107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:41.431639 2107 hybrid_clock.cc:648] HybridClock initialized: now 1781894261431620 us; error 36 us; skew 500 ppm
Jun 19 18:37:41 dist-test-slave-rxv6 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:41.434427 2107 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:37:41.435297 2107 webserver.cc:460] Webserver started at http://127.0.98.62:36087/ using document root <none> and password file <none>
I20260619 18:37:41.435802 2107 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:41.435858 2107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:41.436020 2107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:41.438009 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "4d98dd1fdde74542ba4376f8ab70d2fa"
format_stamp: "Formatted at 2026-06-19 18:37:41 on dist-test-slave-rxv6"
server_key: "22366ea46a204899fd94817061ad9f52"
server_key_iv: "b36694e38e6cb46b32e16cc6e5921a2b"
server_key_version: "encryptionkey@0"
I20260619 18:37:41.438520 2107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d98dd1fdde74542ba4376f8ab70d2fa"
format_stamp: "Formatted at 2026-06-19 18:37:41 on dist-test-slave-rxv6"
server_key: "22366ea46a204899fd94817061ad9f52"
server_key_iv: "b36694e38e6cb46b32e16cc6e5921a2b"
server_key_version: "encryptionkey@0"
I20260619 18:37:41.442250 2107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:37:41.444581 2122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:41.445760 2107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:37:41.445856 2107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "4d98dd1fdde74542ba4376f8ab70d2fa"
format_stamp: "Formatted at 2026-06-19 18:37:41 on dist-test-slave-rxv6"
server_key: "22366ea46a204899fd94817061ad9f52"
server_key_iv: "b36694e38e6cb46b32e16cc6e5921a2b"
server_key_version: "encryptionkey@0"
I20260619 18:37:41.445997 2107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:41.467862 2107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:41.471186 2107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:41.471345 2107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:41.479247 2107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41049
I20260619 18:37:41.479252 2184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41049 every 8 connection(s)
I20260619 18:37:41.480291 2107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:37:41.483206 2185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:41.485219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 2107
I20260619 18:37:41.485308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:37:41.485538 392 external_mini_cluster.cc:1428] Setting key 081c448e400a62b3d7beab5a4b87b578
I20260619 18:37:41.489634 2185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa: Bootstrap starting.
I20260619 18:37:41.492331 2185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:41.493217 2185 log.cc:824] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:37:41 dist-test-slave-rxv6 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:41.495450 2185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa: No bootstrap required, opened a new log
I20260619 18:37:41.499707 2185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d98dd1fdde74542ba4376f8ab70d2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41049 } }
I20260619 18:37:41.500033 2185 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:41.500137 2185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d98dd1fdde74542ba4376f8ab70d2fa, State: Initialized, Role: FOLLOWER
I20260619 18:37:41.500696 2185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d98dd1fdde74542ba4376f8ab70d2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41049 } }
I20260619 18:37:41.500886 2185 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:37:41.500955 2185 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:37:41.501053 2185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:41.501333 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:41.486952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.0.98.62:41049)
0619 18:37:41.487437 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:41.487458 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:41.487504 (+ 46us) server_negotiation.cc:407] Connection header received
0619 18:37:41.488433 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:41.488458 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:41.488901 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:41.489287 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:41.490034 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.491491 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.492432 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.493115 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.493274 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:41.495987 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:41.495993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:41.496012 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:41.496045 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:41.497994 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:41.498490 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:41.498496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:41.498502 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:41.498665 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:41.498952 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:41.498956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:41.498958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:41.499495 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:41.499702 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:41.499966 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 18:37:41.500290 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":145,"threads_started":1}
I20260619 18:37:41.502318 2185 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d98dd1fdde74542ba4376f8ab70d2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41049 } }
I20260619 18:37:41.502700 2185 leader_election.cc:302] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d98dd1fdde74542ba4376f8ab70d2fa; no voters:
I20260619 18:37:41.503062 2185 leader_election.cc:288] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:37:41.503208 2190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:41.503427 2190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [term 1 LEADER]: Becoming Leader. State: Replica: 4d98dd1fdde74542ba4376f8ab70d2fa, State: Running, Role: LEADER
I20260619 18:37:41.503824 2190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d98dd1fdde74542ba4376f8ab70d2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41049 } }
I20260619 18:37:41.504356 2185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:37:41.505007 2190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d98dd1fdde74542ba4376f8ab70d2fa. Latest consensus state: current_term: 1 leader_uuid: "4d98dd1fdde74542ba4376f8ab70d2fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d98dd1fdde74542ba4376f8ab70d2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41049 } } }
I20260619 18:37:41.505141 2190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:41.505594 2191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d98dd1fdde74542ba4376f8ab70d2fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d98dd1fdde74542ba4376f8ab70d2fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41049 } } }
I20260619 18:37:41.505723 2191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:41.505990 2199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:37:41.509229 2199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:37:41.515441 2199 catalog_manager.cc:1347] Generated new cluster ID: 6a149f37a9be4644a04673713a833990
I20260619 18:37:41.515527 2199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:37:41.541324 2199 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:37:41.542457 2199 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:37:41.548157 2199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa: Generated new TSK 0
I20260619 18:37:41.549057 2199 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:37:41.578842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41049
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:41.689378 2211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:41.689669 2211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:41.689718 2211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:41.693882 2211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:41.693977 2211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:41.694020 2211 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 18:37:41.694033 2211 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:37:41.694068 2211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:37:41.699542 2211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:41.700702 2211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:41.701735 2211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:41.711789 2217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:41.711772 2219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:41.711840 2211 server_base.cc:1034] running on GCE node
W20260619 18:37:41.711793 2216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:41.712505 2211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:41.713244 2211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:41.714432 2211 hybrid_clock.cc:648] HybridClock initialized: now 1781894261714397 us; error 46 us; skew 500 ppm
Jun 19 18:37:41 dist-test-slave-rxv6 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:41.717235 2211 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:37:41.718200 2211 webserver.cc:460] Webserver started at http://127.0.98.1:34621/ using document root <none> and password file <none>
I20260619 18:37:41.718808 2211 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:41.718881 2211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:41.719085 2211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:41.721196 2211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "1f8a2e3a2e574a5f8096731a9c40efaf"
format_stamp: "Formatted at 2026-06-19 18:37:41 on dist-test-slave-rxv6"
server_key: "6da2ad529ffea9a2bfbce080496e5acc"
server_key_iv: "91b2f9e4bdfc2aba6e3e916d6eb8a98d"
server_key_version: "encryptionkey@0"
I20260619 18:37:41.721771 2211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1f8a2e3a2e574a5f8096731a9c40efaf"
format_stamp: "Formatted at 2026-06-19 18:37:41 on dist-test-slave-rxv6"
server_key: "6da2ad529ffea9a2bfbce080496e5acc"
server_key_iv: "91b2f9e4bdfc2aba6e3e916d6eb8a98d"
server_key_version: "encryptionkey@0"
I20260619 18:37:41.726048 2211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:37:41.728650 2226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:41.730067 2211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:37:41.730188 2211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "1f8a2e3a2e574a5f8096731a9c40efaf"
format_stamp: "Formatted at 2026-06-19 18:37:41 on dist-test-slave-rxv6"
server_key: "6da2ad529ffea9a2bfbce080496e5acc"
server_key_iv: "91b2f9e4bdfc2aba6e3e916d6eb8a98d"
server_key_version: "encryptionkey@0"
I20260619 18:37:41.730300 2211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:41.747962 2211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:41.751207 2211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:41.751385 2211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:41.752811 2211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:41.752869 2211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:41.752921 2211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:41.752947 2211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:37:41 dist-test-slave-rxv6 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:41.763141 2211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37819
I20260619 18:37:41.763197 2341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37819 every 8 connection(s)
I20260619 18:37:41.764160 2211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:37:41.765269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 2211
I20260619 18:37:41.765370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:37:41.765653 392 external_mini_cluster.cc:1428] Setting key 47888778b5d483889596caaa634470e6
I20260619 18:37:41.765817 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:41.753574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42627 (local address 127.0.98.62:41049)
0619 18:37:41.753729 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:41.753739 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:41.754606 (+ 867us) server_negotiation.cc:407] Connection header received
0619 18:37:41.755878 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:41.755882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:41.755939 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:41.756018 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:41.757315 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.757854 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.759068 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.759348 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.759437 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:41.762198 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:41.762202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:41.762204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:41.762231 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:41.763692 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:41.764254 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:41.764257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:41.764260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:41.764342 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:41.764714 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:41.764735 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:41.764737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:41.764966 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:41.765058 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:41.765525 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 18:37:41.765666 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:37:41.766623 2235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:41.753972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41049 (local address 127.0.98.1:42627)
0619 18:37:41.754434 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:41.754492 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:41.755609 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:41.756224 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:41.756250 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:41.756612 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:41.757151 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:41.757166 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.758009 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.758012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:41.758947 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:41.758957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.759506 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.759510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:41.759660 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:41.760224 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:41.760245 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:41.762025 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:41.763808 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:41.763813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:41.763828 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:41.764129 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:41.764427 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:41.764430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:41.764432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:41.764603 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:41.765074 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:41.765079 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:41.765300 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 18:37:41.765472 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 18:37:41.765737 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
I20260619 18:37:41.767882 2342 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41049
I20260619 18:37:41.768021 2342 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:41.768473 2342 heartbeater.cc:511] Master 127.0.98.62:41049 requested a full tablet report, sending...
I20260619 18:37:41.770191 2138 ts_manager.cc:194] Registered new tserver with Master: 1f8a2e3a2e574a5f8096731a9c40efaf (127.0.98.1:37819)
I20260619 18:37:41.771334 2138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42627
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 18:37:41.782244 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:41.770067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40065 (local address 127.0.98.62:41049)
0619 18:37:41.770222 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:41.770226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:41.775263 (+ 5037us) server_negotiation.cc:407] Connection header received
0619 18:37:41.775328 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:41.775330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:41.775376 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:41.775459 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:41.776088 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.776603 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.777674 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.778006 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.778110 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:41.779357 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:41.779360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:41.779362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:41.779383 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:41.780747 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:41.781120 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:41.781123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:41.781125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:41.781198 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:41.781431 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:41.781434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:41.781436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:41.781620 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:41.781709 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:41.781986 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 18:37:41.782091 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:37:41.782259 2349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:41.770031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41049 (local address 127.0.98.1:40065)
0619 18:37:41.775141 (+ 5110us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:41.775155 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:41.775257 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:41.775517 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:41.775520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:41.775798 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:41.775950 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:41.775958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.776817 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.776822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:41.777558 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:41.777568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.778166 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.778169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:41.778246 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:41.778767 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:41.778782 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:41.779226 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:41.780849 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:41.780852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:41.780855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:41.781034 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:41.781277 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:41.781280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:41.781282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:41.781353 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:41.781713 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:41.781716 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:41.781799 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 18:37:41.781880 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 18:37:41.782066 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5026,"thread_start_us":78,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:37:41.793962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41049
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:41.902093 2351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:41.902364 2351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:41.902410 2351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:41.906572 2351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:41.906649 2351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:41.906692 2351 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 18:37:41.906720 2351 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:37:41.906770 2351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:37:41.912210 2351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:41.913282 2351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:41.914311 2351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:41.923899 2357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:41.923892 2359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:41.923892 2356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:41.924067 2351 server_base.cc:1034] running on GCE node
I20260619 18:37:41.924547 2351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:41.925201 2351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:41.926352 2351 hybrid_clock.cc:648] HybridClock initialized: now 1781894261926333 us; error 31 us; skew 500 ppm
Jun 19 18:37:41 dist-test-slave-rxv6 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:41.929013 2351 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:37:41.929949 2351 webserver.cc:460] Webserver started at http://127.0.98.2:45629/ using document root <none> and password file <none>
I20260619 18:37:41.930486 2351 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:41.930552 2351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:41.930747 2351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:41.932776 2351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "7f472c33a0bf47ac86bf54c950f64f91"
format_stamp: "Formatted at 2026-06-19 18:37:41 on dist-test-slave-rxv6"
server_key: "f3e2f6afdd167bbdc2de66b6e808e573"
server_key_iv: "5eec7bb3e26613a71b059630f9abf4ce"
server_key_version: "encryptionkey@0"
I20260619 18:37:41.933295 2351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7f472c33a0bf47ac86bf54c950f64f91"
format_stamp: "Formatted at 2026-06-19 18:37:41 on dist-test-slave-rxv6"
server_key: "f3e2f6afdd167bbdc2de66b6e808e573"
server_key_iv: "5eec7bb3e26613a71b059630f9abf4ce"
server_key_version: "encryptionkey@0"
I20260619 18:37:41.936995 2351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:37:41.939479 2366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:41.940816 2351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:37:41.940928 2351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "7f472c33a0bf47ac86bf54c950f64f91"
format_stamp: "Formatted at 2026-06-19 18:37:41 on dist-test-slave-rxv6"
server_key: "f3e2f6afdd167bbdc2de66b6e808e573"
server_key_iv: "5eec7bb3e26613a71b059630f9abf4ce"
server_key_version: "encryptionkey@0"
I20260619 18:37:41.941028 2351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:41.968261 2351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:41.971987 2351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:41.972189 2351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:41.973709 2351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:41.973764 2351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:41.973819 2351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:41.973845 2351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:37:41 dist-test-slave-rxv6 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:41.984555 2351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37763
I20260619 18:37:41.984643 2481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37763 every 8 connection(s)
I20260619 18:37:41.985503 2351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:37:41.987303 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:41.974781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54695 (local address 127.0.98.62:41049)
0619 18:37:41.974901 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:41.974904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:41.976076 (+ 1172us) server_negotiation.cc:407] Connection header received
0619 18:37:41.977227 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:41.977230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:41.977289 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:41.977365 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:41.978654 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.979196 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.980480 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.980793 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.980897 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:41.983793 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:41.983796 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:41.983799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:41.983823 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:41.985268 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:41.985801 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:41.985804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:41.985805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:41.985897 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:41.986278 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:41.986282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:41.986283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:41.986508 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:41.986606 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:41.987078 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 18:37:41.987181 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:37:41.988112 2375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:41.975335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41049 (local address 127.0.98.2:54695)
0619 18:37:41.975914 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:41.975958 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:41.976997 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:41.977566 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:41.977574 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:41.977952 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:41.978520 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:41.978535 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.979310 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.979314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:41.980356 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:41.980366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.980964 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.980969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:41.981173 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:41.981786 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:41.981803 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:41.983622 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:41.985377 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:41.985382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:41.985396 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:41.985709 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:41.986027 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:41.986030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:41.986032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:41.986188 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:41.986610 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:41.986615 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:41.986887 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 18:37:41.987039 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 18:37:41.987295 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":158,"threads_started":1}
I20260619 18:37:41.989313 2482 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41049
I20260619 18:37:41.989442 2482 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:41.989974 2482 heartbeater.cc:511] Master 127.0.98.62:41049 requested a full tablet report, sending...
I20260619 18:37:41.990993 2138 ts_manager.cc:194] Registered new tserver with Master: 7f472c33a0bf47ac86bf54c950f64f91 (127.0.98.2:37763)
I20260619 18:37:41.991078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 2351
I20260619 18:37:41.991174 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:37:41.991492 392 external_mini_cluster.cc:1428] Setting key d9c8dc85f73c5197e8f44c9cc222cf59
I20260619 18:37:41.991698 2138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54695
I20260619 18:37:41.998484 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:41.991643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46625 (local address 127.0.98.62:41049)
0619 18:37:41.991746 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:41.991750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:41.991765 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:37:41.991930 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:41.991933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:41.991977 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:41.992032 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:41.992486 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.993046 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.993850 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.994123 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.994210 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:41.995393 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:41.995396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:41.995398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:41.995418 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:41.996722 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:41.997204 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:41.997207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:41.997208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:41.997285 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:41.997660 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:41.997663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:41.997664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:41.997887 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:41.998013 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:41.998276 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 18:37:41.998369 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:37:41.998564 2488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:41.991402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41049 (local address 127.0.98.2:46625)
0619 18:37:41.991711 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:41.991725 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:41.991845 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:41.992058 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:41.992060 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:41.992253 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:41.992379 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:41.992386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.993168 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.993171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:41.993746 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:41.993755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:41.994325 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:41.994328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:41.994396 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:41.994950 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:41.994964 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:41.995285 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:41.996846 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:41.996850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:41.996852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:41.997087 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:41.997377 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:41.997380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:41.997382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:41.997476 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:41.998020 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:41.998023 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:41.998120 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 18:37:41.998227 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 18:37:41.998375 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:37:42.017441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41049
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:42.127104 2491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:42.127377 2491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:42.127422 2491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:42.131528 2491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:42.131599 2491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:42.131641 2491 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 18:37:42.131656 2491 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:37:42.131690 2491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:37:42.137254 2491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:42.138315 2491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:42.139271 2491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:42.148739 2497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:42.148739 2499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:42.148860 2496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:42.149055 2491 server_base.cc:1034] running on GCE node
I20260619 18:37:42.149498 2491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:42.150185 2491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:42.151346 2491 hybrid_clock.cc:648] HybridClock initialized: now 1781894262151318 us; error 41 us; skew 500 ppm
Jun 19 18:37:42 dist-test-slave-rxv6 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:42.154158 2491 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:37:42.155073 2491 webserver.cc:460] Webserver started at http://127.0.98.3:36727/ using document root <none> and password file <none>
I20260619 18:37:42.155608 2491 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:42.155681 2491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:42.155872 2491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:42.157876 2491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "8a2bfe2f746641b6b790a7601cdd66f5"
format_stamp: "Formatted at 2026-06-19 18:37:42 on dist-test-slave-rxv6"
server_key: "63247a2ff3be4b08773bab698942144a"
server_key_iv: "ab9a1ff6d23114320cd96d6bcefc6e2b"
server_key_version: "encryptionkey@0"
I20260619 18:37:42.158452 2491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8a2bfe2f746641b6b790a7601cdd66f5"
format_stamp: "Formatted at 2026-06-19 18:37:42 on dist-test-slave-rxv6"
server_key: "63247a2ff3be4b08773bab698942144a"
server_key_iv: "ab9a1ff6d23114320cd96d6bcefc6e2b"
server_key_version: "encryptionkey@0"
I20260619 18:37:42.162225 2491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:37:42.164609 2506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:42.165776 2491 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:37:42.165892 2491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "8a2bfe2f746641b6b790a7601cdd66f5"
format_stamp: "Formatted at 2026-06-19 18:37:42 on dist-test-slave-rxv6"
server_key: "63247a2ff3be4b08773bab698942144a"
server_key_iv: "ab9a1ff6d23114320cd96d6bcefc6e2b"
server_key_version: "encryptionkey@0"
I20260619 18:37:42.166023 2491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:42.201556 2491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:42.204774 2491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:42.204969 2491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:42.206575 2491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:42.206626 2491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:42.206688 2491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:42.206714 2491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:37:42 dist-test-slave-rxv6 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:42.216917 2491 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43959
I20260619 18:37:42.217036 2621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43959 every 8 connection(s)
I20260619 18:37:42.217902 2491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:37:42.220572 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.207413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50023 (local address 127.0.98.62:41049)
0619 18:37:42.207528 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:42.207532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:42.208586 (+ 1054us) server_negotiation.cc:407] Connection header received
0619 18:37:42.209567 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:42.209572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:42.209635 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:42.209727 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:42.211463 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.212129 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.213793 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.214074 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.214159 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.217256 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:42.217259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:42.217261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:42.217285 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:42.218900 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:42.219312 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:42.219314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:42.219315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:42.219383 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:42.219611 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:42.219613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:42.219614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:42.219812 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:42.219890 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:42.220365 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 18:37:42.220456 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 18:37:42.221403 2515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.207951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41049 (local address 127.0.98.3:50023)
0619 18:37:42.208442 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:42.208484 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:42.209338 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:42.210086 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:42.210095 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:42.210558 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:42.211311 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.211333 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.212243 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.212247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.213684 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.213696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.214354 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.214363 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.214576 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.215327 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:42.215349 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:42.217051 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:42.218978 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:42.218982 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:42.218991 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:42.219226 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:42.219437 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:42.219439 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:42.219440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:42.219552 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:42.219926 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:42.219931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:42.220171 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 18:37:42.220335 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 18:37:42.220594 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":141,"threads_started":1}
I20260619 18:37:42.222694 2622 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41049
I20260619 18:37:42.222833 2622 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:42.223346 2622 heartbeater.cc:511] Master 127.0.98.62:41049 requested a full tablet report, sending...
I20260619 18:37:42.224458 2138 ts_manager.cc:194] Registered new tserver with Master: 8a2bfe2f746641b6b790a7601cdd66f5 (127.0.98.3:43959)
I20260619 18:37:42.224609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 2491
I20260619 18:37:42.224700 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:37:42.224994 392 external_mini_cluster.cc:1428] Setting key 490e5005d99461225d118143a3683e60
I20260619 18:37:42.225142 2138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50023
I20260619 18:37:42.227113 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:37:42.231963 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.224607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56069 (local address 127.0.98.62:41049)
0619 18:37:42.224773 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:42.224777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:42.225480 (+ 703us) server_negotiation.cc:407] Connection header received
0619 18:37:42.225651 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:42.225654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:42.225696 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:42.225757 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:42.226263 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.226747 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.227803 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.228012 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.228099 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.229130 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:42.229134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:42.229137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:42.229160 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:42.230490 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:42.230915 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:42.230919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:42.230921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:42.230992 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:42.231229 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:42.231232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:42.231234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:42.231401 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:42.231484 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:42.231717 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 18:37:42.231815 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 18:37:42.231983 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.224543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41049 (local address 127.0.98.3:56069)
0619 18:37:42.225400 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:42.225416 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:42.225545 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:42.225794 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:42.225797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:42.226030 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:42.226152 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.226160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.226861 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.226864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.227674 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.227685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.228115 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.228118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.228209 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.228669 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:42.228689 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:42.229005 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:42.230589 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:42.230593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:42.230595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:42.230829 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:42.231069 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:42.231072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:42.231074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:42.231150 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:42.231486 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:42.231489 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:42.231574 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 18:37:42.231656 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 18:37:42.231803 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":91,"threads_started":1}
Jun 19 18:37:42 dist-test-slave-rxv6 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:37:42 dist-test-slave-rxv6 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:42.252084 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.243952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50840 (local address 127.0.98.62:41049)
0619 18:37:42.244073 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:42.244077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:42.244089 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:37:42.244126 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:42.244128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:42.244166 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:42.244232 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:42.244942 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.245508 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.246372 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.246584 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.246670 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.249031 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:42.249034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:42.249036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:42.249064 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:42.250578 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:42.250987 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:42.250989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:42.250991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:42.251069 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:42.251300 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:42.251301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:42.251303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:42.251474 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:42.251574 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:42.251844 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 18:37:42.251960 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:37:42.255151 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:37:42.258144 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:37:42.272634 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.267318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58336 (local address 127.0.98.2:37763)
0619 18:37:42.267981 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:42.268027 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:42.268051 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:37:42.268127 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:42.268134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:42.268351 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:42.268502 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:42.268799 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.269820 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.271441 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.271987 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.272087 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.272275 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:42.272392 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 18:37:42.272483 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":97,"threads_started":1}
I20260619 18:37:42.272626 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.266607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37763 (local address 127.0.0.1:58336)
0619 18:37:42.266943 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:42.266999 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:42.267284 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:42.268507 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:42.268514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:42.268536 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:42.268666 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.268674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.269978 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.269981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.271303 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.271314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.272105 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.272109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.272214 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.272251 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 18:37:42.272353 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 18:37:42.272476 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":107,"thread_start_us":79,"threads_started":1}
I20260619 18:37:42.273458 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.266710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43959 (local address 127.0.0.1:59768)
0619 18:37:42.267560 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:42.267572 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:42.267647 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:42.268655 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:42.268659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:42.268683 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:42.268797 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.268803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.270282 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.270285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.271313 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.271320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.272546 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.272550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.272638 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.272661 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:37:42.273094 (+ 433us) client_negotiation.cc:240] Negotiation successful
0619 18:37:42.273171 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":91,"spinlock_wait_cycles":8832,"thread_start_us":84,"threads_started":1}
I20260619 18:37:42.273782 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.267866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54802 (local address 127.0.98.1:37819)
0619 18:37:42.268202 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:42.268207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:42.268225 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:37:42.268283 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:42.268287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:42.268431 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:42.268567 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:42.268925 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.270154 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.272241 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.273085 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.273189 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.273330 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:42.273542 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 18:37:42.273639 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":83,"threads_started":1}
I20260619 18:37:42.273797 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.267332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37819 (local address 127.0.0.1:54802)
0619 18:37:42.267669 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:42.267684 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:42.267819 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:42.268655 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:42.268659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:42.268689 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:42.268797 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.268803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.270282 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.270285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.272098 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.272109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.273223 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.273227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.273310 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.273332 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:37:42.273401 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 18:37:42.273466 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":813,"spinlock_wait_cycles":7936,"thread_start_us":89,"threads_started":1}
I20260619 18:37:42.275682 2416 tablet_service.cc:1467] Processing CreateTablet for tablet f5cb3dc0e2ac48c7a357e4988bd6bd85 (DEFAULT_TABLE table=test-table [id=840f2d08ff3249e0a9a7168803112ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:42.276582 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.267089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59768 (local address 127.0.98.3:43959)
0619 18:37:42.268092 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:42.268102 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:42.268120 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:37:42.268179 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:42.268182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:42.268346 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:42.268502 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:42.268923 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.270160 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.271425 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.272228 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.272327 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.272485 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:42.276332 (+ 3847us) server_negotiation.cc:299] Negotiation successful
0619 18:37:42.276440 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":116,"threads_started":1}
I20260619 18:37:42.276228 2276 tablet_service.cc:1467] Processing CreateTablet for tablet f5cb3dc0e2ac48c7a357e4988bd6bd85 (DEFAULT_TABLE table=test-table [id=840f2d08ff3249e0a9a7168803112ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:42.276656 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5cb3dc0e2ac48c7a357e4988bd6bd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:42.277273 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5cb3dc0e2ac48c7a357e4988bd6bd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:42.279146 2556 tablet_service.cc:1467] Processing CreateTablet for tablet f5cb3dc0e2ac48c7a357e4988bd6bd85 (DEFAULT_TABLE table=test-table [id=840f2d08ff3249e0a9a7168803112ae4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:42.280177 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5cb3dc0e2ac48c7a357e4988bd6bd85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:42.283149 2645 tablet_bootstrap.cc:492] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf: Bootstrap starting.
I20260619 18:37:42.284976 2646 tablet_bootstrap.cc:492] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91: Bootstrap starting.
I20260619 18:37:42.285511 2645 tablet_bootstrap.cc:654] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:42.286530 2645 log.cc:824] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:42.287037 2646 tablet_bootstrap.cc:654] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:42.287591 2648 tablet_bootstrap.cc:492] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5: Bootstrap starting.
I20260619 18:37:42.287817 2646 log.cc:824] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:42.288364 2645 tablet_bootstrap.cc:492] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf: No bootstrap required, opened a new log
I20260619 18:37:42.288563 2645 ts_tablet_manager.cc:1397] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:37:42.289479 2646 tablet_bootstrap.cc:492] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91: No bootstrap required, opened a new log
I20260619 18:37:42.289649 2646 ts_tablet_manager.cc:1397] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:37:42.289700 2648 tablet_bootstrap.cc:654] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:42.290555 2648 log.cc:824] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:42.292222 2645 raft_consensus.cc:348] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } }
I20260619 18:37:42.292439 2645 raft_consensus.cc:374] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:42.292474 2645 raft_consensus.cc:729] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f8a2e3a2e574a5f8096731a9c40efaf, State: Initialized, Role: FOLLOWER
I20260619 18:37:42.292738 2646 raft_consensus.cc:348] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } }
I20260619 18:37:42.292881 2645 consensus_queue.cc:260] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } }
I20260619 18:37:42.292949 2646 raft_consensus.cc:374] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:42.292999 2646 raft_consensus.cc:729] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f472c33a0bf47ac86bf54c950f64f91, State: Initialized, Role: FOLLOWER
I20260619 18:37:42.293380 2646 consensus_queue.cc:260] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } }
I20260619 18:37:42.293512 2342 heartbeater.cc:503] Master 127.0.98.62:41049 was elected leader, sending a full tablet report...
I20260619 18:37:42.293860 2645 ts_tablet_manager.cc:1428] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 18:37:42.294111 2482 heartbeater.cc:503] Master 127.0.98.62:41049 was elected leader, sending a full tablet report...
I20260619 18:37:42.294364 2646 ts_tablet_manager.cc:1428] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 18:37:42.295861 2648 tablet_bootstrap.cc:492] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5: No bootstrap required, opened a new log
I20260619 18:37:42.296064 2648 ts_tablet_manager.cc:1397] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260619 18:37:42.299630 2648 raft_consensus.cc:348] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } }
I20260619 18:37:42.299863 2648 raft_consensus.cc:374] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:42.299903 2648 raft_consensus.cc:729] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2bfe2f746641b6b790a7601cdd66f5, State: Initialized, Role: FOLLOWER
I20260619 18:37:42.300266 2648 consensus_queue.cc:260] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } }
I20260619 18:37:42.300954 2622 heartbeater.cc:503] Master 127.0.98.62:41049 was elected leader, sending a full tablet report...
I20260619 18:37:42.301158 2648 ts_tablet_manager.cc:1428] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 18:37:42.371683 2653 raft_consensus.cc:482] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:37:42.371893 2653 raft_consensus.cc:504] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } }
I20260619 18:37:42.373000 2653 leader_election.cc:288] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f472c33a0bf47ac86bf54c950f64f91 (127.0.98.2:37763), 1f8a2e3a2e574a5f8096731a9c40efaf (127.0.98.1:37819)
I20260619 18:37:42.377254 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.373342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39369 (local address 127.0.98.1:37819)
0619 18:37:42.373480 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:42.373484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:42.373542 (+ 58us) server_negotiation.cc:407] Connection header received
0619 18:37:42.373721 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:42.373724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:42.373819 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:42.373926 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:42.374209 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.374961 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.376187 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.376778 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.376853 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.376908 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:42.377098 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 18:37:42.377150 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:37:42.377300 2515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.373251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37819 (local address 127.0.98.3:39369)
0619 18:37:42.373479 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:42.373493 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:42.373600 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:42.373943 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:42.373946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:42.373968 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:42.374101 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.374108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.375060 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.375063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.376070 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.376079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.376882 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.376886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.376987 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.377012 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:37:42.377083 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 18:37:42.377151 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"spinlock_wait_cycles":4864}
I20260619 18:37:42.377871 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.373363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42643 (local address 127.0.98.2:37763)
0619 18:37:42.373483 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:42.373487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:42.373984 (+ 497us) server_negotiation.cc:407] Connection header received
0619 18:37:42.374080 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:42.374083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:42.374132 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:42.374202 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:42.374888 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.375582 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.376791 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.377339 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.377432 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.377489 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:42.377720 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 18:37:42.377784 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:37:42.377871 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.373252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37763 (local address 127.0.98.3:42643)
0619 18:37:42.373875 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:42.373889 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:42.373997 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:42.374213 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:42.374215 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:42.374230 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:42.374771 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.374780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.375687 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.375689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.376661 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.376670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.377485 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.377489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.377587 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.377609 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:37:42.377674 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 18:37:42.377731 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":246,"thread_start_us":146,"threads_started":1}
I20260619 18:37:42.378099 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5cb3dc0e2ac48c7a357e4988bd6bd85" candidate_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" is_pre_election: true
I20260619 18:37:42.378424 2296 raft_consensus.cc:2454] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2bfe2f746641b6b790a7601cdd66f5 in term 0.
I20260619 18:37:42.378567 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5cb3dc0e2ac48c7a357e4988bd6bd85" candidate_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f472c33a0bf47ac86bf54c950f64f91" is_pre_election: true
I20260619 18:37:42.378849 2436 raft_consensus.cc:2454] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2bfe2f746641b6b790a7601cdd66f5 in term 0.
I20260619 18:37:42.378935 2508 leader_election.cc:302] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f8a2e3a2e574a5f8096731a9c40efaf, 8a2bfe2f746641b6b790a7601cdd66f5; no voters:
I20260619 18:37:42.379225 2653 raft_consensus.cc:2781] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:37:42.379300 2653 raft_consensus.cc:482] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:37:42.379330 2653 raft_consensus.cc:3037] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:42.380323 2653 raft_consensus.cc:504] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } }
I20260619 18:37:42.380657 2653 leader_election.cc:288] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [CANDIDATE]: Term 1 election: Requested vote from peers 7f472c33a0bf47ac86bf54c950f64f91 (127.0.98.2:37763), 1f8a2e3a2e574a5f8096731a9c40efaf (127.0.98.1:37819)
I20260619 18:37:42.380983 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5cb3dc0e2ac48c7a357e4988bd6bd85" candidate_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f472c33a0bf47ac86bf54c950f64f91"
I20260619 18:37:42.380976 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5cb3dc0e2ac48c7a357e4988bd6bd85" candidate_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf"
I20260619 18:37:42.381094 2296 raft_consensus.cc:3037] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:42.381094 2436 raft_consensus.cc:3037] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:42.382362 2296 raft_consensus.cc:2454] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2bfe2f746641b6b790a7601cdd66f5 in term 1.
I20260619 18:37:42.382362 2436 raft_consensus.cc:2454] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2bfe2f746641b6b790a7601cdd66f5 in term 1.
I20260619 18:37:42.382694 2508 leader_election.cc:302] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1f8a2e3a2e574a5f8096731a9c40efaf, 7f472c33a0bf47ac86bf54c950f64f91, 8a2bfe2f746641b6b790a7601cdd66f5; no voters:
I20260619 18:37:42.382874 2653 raft_consensus.cc:2781] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:42.383080 2653 raft_consensus.cc:686] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 LEADER]: Becoming Leader. State: Replica: 8a2bfe2f746641b6b790a7601cdd66f5, State: Running, Role: LEADER
I20260619 18:37:42.383379 2653 consensus_queue.cc:237] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } }
I20260619 18:37:42.386386 2137 catalog_manager.cc:5495] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a2bfe2f746641b6b790a7601cdd66f5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } health_report { overall_health: UNKNOWN } } }
Jun 19 18:37:42 dist-test-slave-rxv6 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894261, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:42.443655 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.435585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41049 (local address 127.0.0.1:50850)
0619 18:37:42.435853 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:42.435869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:42.435993 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:42.436298 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:42.436300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:42.436743 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:42.436837 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.436843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.437586 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.437588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.438220 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.438229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.438658 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.438662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.438736 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.439259 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:42.439270 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:42.440609 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:42.442084 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:42.442093 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:42.442099 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:42.442383 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:42.442664 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:42.442666 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:42.442667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:42.442741 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:42.443141 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:42.443146 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:42.443263 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 18:37:42.443364 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 18:37:42.443489 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":92,"threads_started":1}
I20260619 18:37:42.443674 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.435624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.0.98.62:41049)
0619 18:37:42.435743 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:42.435747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:42.435966 (+ 219us) server_negotiation.cc:407] Connection header received
0619 18:37:42.436087 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:42.436089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:42.436151 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:42.436250 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:42.436959 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.437481 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.438322 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.438530 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.438623 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.440726 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:42.440730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:42.440732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:42.440752 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:42.441995 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:42.442494 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:42.442497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:42.442498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:42.442590 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:42.442850 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:42.442854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:42.442855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:42.443053 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:42.443147 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:42.443415 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 18:37:42.443550 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":42}
I20260619 18:37:42.447615 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50850:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 18:37:42.448328 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:37:42.452934 2276 tablet_service.cc:1467] Processing CreateTablet for tablet 7be12542b1bb4bbeb9bc0caf9c149e72 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f7eb1f2536ea426c890054229a9650ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:37:42.453238 2416 tablet_service.cc:1467] Processing CreateTablet for tablet 7be12542b1bb4bbeb9bc0caf9c149e72 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f7eb1f2536ea426c890054229a9650ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:37:42.453125 2556 tablet_service.cc:1467] Processing CreateTablet for tablet 7be12542b1bb4bbeb9bc0caf9c149e72 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f7eb1f2536ea426c890054229a9650ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:37:42.453594 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be12542b1bb4bbeb9bc0caf9c149e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:42.453794 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be12542b1bb4bbeb9bc0caf9c149e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:42.453794 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7be12542b1bb4bbeb9bc0caf9c149e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:42.456027 2645 tablet_bootstrap.cc:492] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf: Bootstrap starting.
I20260619 18:37:42.456573 2646 tablet_bootstrap.cc:492] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91: Bootstrap starting.
I20260619 18:37:42.456935 2645 tablet_bootstrap.cc:654] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:42.457465 2646 tablet_bootstrap.cc:654] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:42.457841 2648 tablet_bootstrap.cc:492] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5: Bootstrap starting.
I20260619 18:37:42.458324 2645 tablet_bootstrap.cc:492] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf: No bootstrap required, opened a new log
I20260619 18:37:42.458408 2645 ts_tablet_manager.cc:1397] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260619 18:37:42.458621 2648 tablet_bootstrap.cc:654] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:42.458834 2645 raft_consensus.cc:348] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.458931 2645 raft_consensus.cc:374] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:42.458962 2645 raft_consensus.cc:729] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f8a2e3a2e574a5f8096731a9c40efaf, State: Initialized, Role: FOLLOWER
I20260619 18:37:42.459082 2645 consensus_queue.cc:260] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.459381 2645 ts_tablet_manager.cc:1428] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 18:37:42.459342 2651 tablet_replica.cc:439] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } }
I20260619 18:37:42.459478 2651 tablet_replica.cc:442] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:37:42.460012 2646 tablet_bootstrap.cc:492] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91: No bootstrap required, opened a new log
I20260619 18:37:42.460103 2646 ts_tablet_manager.cc:1397] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260619 18:37:42.460143 2648 tablet_bootstrap.cc:492] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5: No bootstrap required, opened a new log
I20260619 18:37:42.460216 2648 ts_tablet_manager.cc:1397] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:37:42.460551 2646 raft_consensus.cc:348] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.460657 2646 raft_consensus.cc:374] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:42.460685 2646 raft_consensus.cc:729] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f472c33a0bf47ac86bf54c950f64f91, State: Initialized, Role: FOLLOWER
I20260619 18:37:42.460645 2648 raft_consensus.cc:348] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.460788 2648 raft_consensus.cc:374] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:42.460831 2648 raft_consensus.cc:729] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2bfe2f746641b6b790a7601cdd66f5, State: Initialized, Role: FOLLOWER
I20260619 18:37:42.460805 2646 consensus_queue.cc:260] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.460951 2648 consensus_queue.cc:260] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.461046 2646 ts_tablet_manager.cc:1428] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 18:37:42.461194 2648 ts_tablet_manager.cc:1428] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:37:42.461053 2652 tablet_replica.cc:439] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } }
I20260619 18:37:42.461264 2652 tablet_replica.cc:442] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:37:42.461269 2653 tablet_replica.cc:439] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } }
I20260619 18:37:42.461376 2653 tablet_replica.cc:442] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5: This TxnStatusTablet replica's current role is: FOLLOWER
W20260619 18:37:42.468827 2623 tablet.cc:2369] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:37:42.486662 2483 tablet.cc:2369] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:37:42.512228 2653 raft_consensus.cc:482] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:37:42.512357 2653 raft_consensus.cc:504] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.512758 2653 leader_election.cc:288] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f8a2e3a2e574a5f8096731a9c40efaf (127.0.98.1:37819), 7f472c33a0bf47ac86bf54c950f64f91 (127.0.98.2:37763)
I20260619 18:37:42.513228 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be12542b1bb4bbeb9bc0caf9c149e72" candidate_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" is_pre_election: true
I20260619 18:37:42.513221 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be12542b1bb4bbeb9bc0caf9c149e72" candidate_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f472c33a0bf47ac86bf54c950f64f91" is_pre_election: true
I20260619 18:37:42.513367 2296 raft_consensus.cc:2454] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2bfe2f746641b6b790a7601cdd66f5 in term 0.
I20260619 18:37:42.513379 2436 raft_consensus.cc:2454] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2bfe2f746641b6b790a7601cdd66f5 in term 0.
I20260619 18:37:42.513667 2510 leader_election.cc:302] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f472c33a0bf47ac86bf54c950f64f91, 8a2bfe2f746641b6b790a7601cdd66f5; no voters:
I20260619 18:37:42.513834 2653 raft_consensus.cc:2781] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:37:42.513888 2653 raft_consensus.cc:482] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:37:42.513939 2653 raft_consensus.cc:3037] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:42.514883 2653 raft_consensus.cc:504] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.515180 2653 leader_election.cc:288] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [CANDIDATE]: Term 1 election: Requested vote from peers 1f8a2e3a2e574a5f8096731a9c40efaf (127.0.98.1:37819), 7f472c33a0bf47ac86bf54c950f64f91 (127.0.98.2:37763)
W20260619 18:37:42.515501 2343 tablet.cc:2369] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:37:42.515581 2436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be12542b1bb4bbeb9bc0caf9c149e72" candidate_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f472c33a0bf47ac86bf54c950f64f91"
I20260619 18:37:42.515581 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7be12542b1bb4bbeb9bc0caf9c149e72" candidate_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf"
I20260619 18:37:42.515697 2296 raft_consensus.cc:3037] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:42.515697 2436 raft_consensus.cc:3037] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:42.516718 2296 raft_consensus.cc:2454] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2bfe2f746641b6b790a7601cdd66f5 in term 1.
I20260619 18:37:42.516718 2436 raft_consensus.cc:2454] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2bfe2f746641b6b790a7601cdd66f5 in term 1.
I20260619 18:37:42.516990 2510 leader_election.cc:302] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1f8a2e3a2e574a5f8096731a9c40efaf, 7f472c33a0bf47ac86bf54c950f64f91, 8a2bfe2f746641b6b790a7601cdd66f5; no voters:
I20260619 18:37:42.517156 2653 raft_consensus.cc:2781] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:42.517257 2653 raft_consensus.cc:686] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 LEADER]: Becoming Leader. State: Replica: 8a2bfe2f746641b6b790a7601cdd66f5, State: Running, Role: LEADER
I20260619 18:37:42.517407 2653 consensus_queue.cc:237] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.518153 2655 tablet_replica.cc:439] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5: TxnStatusTablet state changed. Reason: New leader 8a2bfe2f746641b6b790a7601cdd66f5. Latest consensus state: current_term: 1 leader_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } }
I20260619 18:37:42.518263 2655 tablet_replica.cc:442] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:37:42.518553 2667 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 18:37:42.518847 2667 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 18:37:42.519124 2137 catalog_manager.cc:5495] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a2bfe2f746641b6b790a7601cdd66f5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } health_report { overall_health: HEALTHY } } }
I20260619 18:37:42.544468 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.540583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43959 (local address 127.0.0.1:59784)
0619 18:37:42.540738 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:42.540754 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:42.540866 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:42.541178 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:42.541181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:42.541196 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:37:42.541296 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.541301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.542108 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.542112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.542882 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.542893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.543334 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.543337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.543421 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.543465 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:37:42.544164 (+ 699us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:37:42.544180 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 18:37:42.544242 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 18:37:42.544324 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 18:37:42.544468 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.540695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59784 (local address 127.0.98.3:43959)
0619 18:37:42.540810 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:42.540815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:42.540828 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:37:42.541015 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:42.541017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:42.541071 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:42.541153 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:37:42.541398 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.541999 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.542996 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.543233 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.543327 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.543604 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:37:42.544082 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:37:42.544183 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:42.544252 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 18:37:42.544332 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:37:42.549386 2296 raft_consensus.cc:1261] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 1 FOLLOWER]: Refusing update from remote peer 8a2bfe2f746641b6b790a7601cdd66f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:42.549417 2436 raft_consensus.cc:1261] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 FOLLOWER]: Refusing update from remote peer 8a2bfe2f746641b6b790a7601cdd66f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:42.550055 2655 consensus_queue.cc:1035] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:42.550221 2653 consensus_queue.cc:1035] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:42.553442 2652 tablet_replica.cc:439] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91: TxnStatusTablet state changed. Reason: New leader 8a2bfe2f746641b6b790a7601cdd66f5. Latest consensus state: current_term: 1 leader_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } }
I20260619 18:37:42.553556 2652 tablet_replica.cc:442] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:37:42.553798 2651 tablet_replica.cc:439] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf: TxnStatusTablet state changed. Reason: New leader 8a2bfe2f746641b6b790a7601cdd66f5. Latest consensus state: current_term: 1 leader_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } }
I20260619 18:37:42.553962 2651 tablet_replica.cc:442] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:37:42.557601 2655 tablet_replica.cc:439] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } }
I20260619 18:37:42.557719 2655 tablet_replica.cc:442] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:37:42.558221 2670 mvcc.cc:205] Tried to move back new op lower bound from 7298638899395731456 to 7298638899272601600. Current Snapshot: MvccSnapshot[applied={T|T < 7298638899395731456}]
I20260619 18:37:42.559331 2653 tablet_replica.cc:439] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } }
I20260619 18:37:42.559391 2672 mvcc.cc:205] Tried to move back new op lower bound from 7298638899395731456 to 7298638899272601600. Current Snapshot: MvccSnapshot[applied={T|T < 7298638899395731456}]
I20260619 18:37:42.559437 2653 tablet_replica.cc:442] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 8a2bfe2f746641b6b790a7601cdd66f5: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:37:42.559626 2652 tablet_replica.cc:439] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } }
I20260619 18:37:42.559719 2652 tablet_replica.cc:442] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 7f472c33a0bf47ac86bf54c950f64f91: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:37:42.561273 2651 tablet_replica.cc:439] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } } }
I20260619 18:37:42.561398 2651 tablet_replica.cc:442] T 7be12542b1bb4bbeb9bc0caf9c149e72 P 1f8a2e3a2e574a5f8096731a9c40efaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:37:42.567543 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.564419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59792 (local address 127.0.98.3:43959)
0619 18:37:42.564768 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:42.564771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:42.564780 (+ 9us) server_negotiation.cc:407] Connection header received
0619 18:37:42.564821 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:42.564824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:42.564858 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:42.564914 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:37:42.565172 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.565718 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.566599 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.566815 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.566898 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.567146 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:37:42.567211 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:37:42.567265 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:42.567369 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 18:37:42.567418 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260619 18:37:42.577553 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.574191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43959 (local address 127.0.98.3:47565)
0619 18:37:42.574320 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:42.574335 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:42.574407 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:42.574617 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:42.574619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:42.574624 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:37:42.574726 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.574730 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.575664 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.575666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.576307 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.576317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.576878 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.576887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.576971 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.577011 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:37:42.577283 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:37:42.577287 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 18:37:42.577334 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 18:37:42.577393 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":3}
I20260619 18:37:42.577570 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.574253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47565 (local address 127.0.98.3:43959)
0619 18:37:42.574366 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:42.574370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:42.574384 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:37:42.574485 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:42.574488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:42.574533 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:42.574610 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:37:42.574801 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.575555 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.576422 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.576749 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.576841 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.577143 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:37:42.577217 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:37:42.577278 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:42.577344 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 18:37:42.577392 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":50}
I20260619 18:37:42.584641 2296 raft_consensus.cc:1261] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 1 FOLLOWER]: Refusing update from remote peer 8a2bfe2f746641b6b790a7601cdd66f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:42.584724 2436 raft_consensus.cc:1261] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 FOLLOWER]: Refusing update from remote peer 8a2bfe2f746641b6b790a7601cdd66f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:42.585045 2677 consensus_queue.cc:1035] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:42.585209 2655 consensus_queue.cc:1035] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:42.592389 2671 mvcc.cc:205] Tried to move back new op lower bound from 7298638899543150592 to 7298638898724716544. Current Snapshot: MvccSnapshot[applied={T|T < 7298638899543150592}]
I20260619 18:37:42.625633 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:37:42.625844 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:37:42.628343 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa: Sending DeleteTablet for 3 replicas of tablet f5cb3dc0e2ac48c7a357e4988bd6bd85
I20260619 18:37:42.629074 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet f5cb3dc0e2ac48c7a357e4988bd6bd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58336
I20260619 18:37:42.629159 2556 tablet_service.cc:1514] Processing DeleteTablet for tablet f5cb3dc0e2ac48c7a357e4988bd6bd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59768
I20260619 18:37:42.629626 2686 tablet_replica.cc:330] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91: stopping tablet replica
I20260619 18:37:42.629706 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 18:37:42.629981 2686 raft_consensus.cc:2229] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:37:42.629992 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet f5cb3dc0e2ac48c7a357e4988bd6bd85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54802
W20260619 18:37:42.630143 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:37:42.630195 2686 raft_consensus.cc:2258] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:42.630692 2687 tablet_replica.cc:330] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5: stopping tablet replica
I20260619 18:37:42.630980 2688 tablet_replica.cc:330] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf: stopping tablet replica
I20260619 18:37:42.631333 2688 raft_consensus.cc:2229] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:37:42.631639 2686 ts_tablet_manager.cc:1905] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:42.631685 2687 raft_consensus.cc:2229] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:37:42.632170 2687 raft_consensus.cc:2258] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:42.632449 2688 raft_consensus.cc:2258] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:42.634858 2686 ts_tablet_manager.cc:1918] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:37:42.634969 2686 log.cc:1196] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/f5cb3dc0e2ac48c7a357e4988bd6bd85
I20260619 18:37:42.635568 2687 ts_tablet_manager.cc:1905] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:42.635550 2416 tablet_service.cc:1467] Processing CreateTablet for tablet 6849b44d33654463b1337994af1689e5 (DEFAULT_TABLE table=test-table [id=9535546bf2404b29bb1382cd71b9ef33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:42.635699 2276 tablet_service.cc:1467] Processing CreateTablet for tablet 6849b44d33654463b1337994af1689e5 (DEFAULT_TABLE table=test-table [id=9535546bf2404b29bb1382cd71b9ef33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:42.635800 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6849b44d33654463b1337994af1689e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:42.635955 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6849b44d33654463b1337994af1689e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:42.636263 2688 ts_tablet_manager.cc:1905] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:42.638098 2646 tablet_bootstrap.cc:492] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91: Bootstrap starting.
I20260619 18:37:42.638459 2556 tablet_service.cc:1467] Processing CreateTablet for tablet 6849b44d33654463b1337994af1689e5 (DEFAULT_TABLE table=test-table [id=9535546bf2404b29bb1382cd71b9ef33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:42.638710 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6849b44d33654463b1337994af1689e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:42.639045 2646 tablet_bootstrap.cc:654] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:42.640123 2686 ts_tablet_manager.cc:1939] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 7f472c33a0bf47ac86bf54c950f64f91: Deleting consensus metadata
I20260619 18:37:42.640964 2645 tablet_bootstrap.cc:492] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf: Bootstrap starting.
I20260619 18:37:42.641460 2688 ts_tablet_manager.cc:1918] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:37:42.641476 2126 catalog_manager.cc:4841] TS 7f472c33a0bf47ac86bf54c950f64f91 (127.0.98.2:37763): tablet f5cb3dc0e2ac48c7a357e4988bd6bd85 (table test-table [id=840f2d08ff3249e0a9a7168803112ae4]) successfully deleted
I20260619 18:37:42.642031 2648 tablet_bootstrap.cc:492] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5: Bootstrap starting.
I20260619 18:37:42.642172 2688 log.cc:1196] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/wal/wals/f5cb3dc0e2ac48c7a357e4988bd6bd85
I20260619 18:37:42.642459 2646 tablet_bootstrap.cc:492] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91: No bootstrap required, opened a new log
I20260619 18:37:42.642557 2646 ts_tablet_manager.cc:1397] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260619 18:37:42.642691 2687 ts_tablet_manager.cc:1918] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:37:42.642849 2687 log.cc:1196] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/f5cb3dc0e2ac48c7a357e4988bd6bd85
I20260619 18:37:42.642971 2688 ts_tablet_manager.cc:1939] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 1f8a2e3a2e574a5f8096731a9c40efaf: Deleting consensus metadata
I20260619 18:37:42.642936 2648 tablet_bootstrap.cc:654] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:42.643035 2646 raft_consensus.cc:348] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.643159 2646 raft_consensus.cc:374] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:42.643193 2646 raft_consensus.cc:729] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f472c33a0bf47ac86bf54c950f64f91, State: Initialized, Role: FOLLOWER
I20260619 18:37:42.643332 2646 consensus_queue.cc:260] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.643606 2646 ts_tablet_manager.cc:1428] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260619 18:37:42.643473 2645 tablet_bootstrap.cc:654] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:42.645152 2687 ts_tablet_manager.cc:1939] T f5cb3dc0e2ac48c7a357e4988bd6bd85 P 8a2bfe2f746641b6b790a7601cdd66f5: Deleting consensus metadata
I20260619 18:37:42.645551 2645 tablet_bootstrap.cc:492] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf: No bootstrap required, opened a new log
I20260619 18:37:42.645643 2645 ts_tablet_manager.cc:1397] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260619 18:37:42.646090 2645 raft_consensus.cc:348] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.646211 2645 raft_consensus.cc:374] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:42.646263 2645 raft_consensus.cc:729] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f8a2e3a2e574a5f8096731a9c40efaf, State: Initialized, Role: FOLLOWER
I20260619 18:37:42.646386 2124 catalog_manager.cc:4841] TS 8a2bfe2f746641b6b790a7601cdd66f5 (127.0.98.3:43959): tablet f5cb3dc0e2ac48c7a357e4988bd6bd85 (table test-table [id=840f2d08ff3249e0a9a7168803112ae4]) successfully deleted
I20260619 18:37:42.646390 2645 consensus_queue.cc:260] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.646639 2645 ts_tablet_manager.cc:1428] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 18:37:42.646700 2124 catalog_manager.cc:4841] TS 1f8a2e3a2e574a5f8096731a9c40efaf (127.0.98.1:37819): tablet f5cb3dc0e2ac48c7a357e4988bd6bd85 (table test-table [id=840f2d08ff3249e0a9a7168803112ae4]) successfully deleted
I20260619 18:37:42.647460 2648 tablet_bootstrap.cc:492] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5: No bootstrap required, opened a new log
I20260619 18:37:42.647540 2648 ts_tablet_manager.cc:1397] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260619 18:37:42.647997 2648 raft_consensus.cc:348] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.648113 2648 raft_consensus.cc:374] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:42.648144 2648 raft_consensus.cc:729] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2bfe2f746641b6b790a7601cdd66f5, State: Initialized, Role: FOLLOWER
I20260619 18:37:42.648269 2648 consensus_queue.cc:260] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.648523 2648 ts_tablet_manager.cc:1428] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 18:37:42.701401 2652 raft_consensus.cc:482] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:37:42.701547 2652 raft_consensus.cc:504] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.702708 2652 leader_election.cc:288] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f8a2e3a2e574a5f8096731a9c40efaf (127.0.98.1:37819), 8a2bfe2f746641b6b790a7601cdd66f5 (127.0.98.3:43959)
I20260619 18:37:42.707427 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.702970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37819 (local address 127.0.98.2:33911)
0619 18:37:42.703262 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:42.703275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:42.703445 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:42.703827 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:42.703830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:42.703849 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:42.703968 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.703975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.704833 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.704837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.706112 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.706123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.707003 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.707008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.707096 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.707119 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:37:42.707190 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 18:37:42.707259 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":94,"threads_started":1}
I20260619 18:37:42.707429 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.703054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33911 (local address 127.0.98.1:37819)
0619 18:37:42.703181 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:42.703185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:42.703360 (+ 175us) server_negotiation.cc:407] Connection header received
0619 18:37:42.703576 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:42.703579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:42.703631 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:42.703680 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:42.704063 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.704731 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.706242 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.706861 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.706972 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.707040 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:42.707210 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 18:37:42.707298 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:37:42.707561 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.703484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44849 (local address 127.0.98.3:43959)
0619 18:37:42.703602 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:42.703606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:42.703931 (+ 325us) server_negotiation.cc:407] Connection header received
0619 18:37:42.704021 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:42.704024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:42.704079 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:42.704150 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:42.704500 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.705269 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.706540 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.706938 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.707036 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.707096 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:42.707371 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 18:37:42.707446 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:37:42.708036 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.703373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43959 (local address 127.0.98.2:44849)
0619 18:37:42.703827 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:42.703840 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:42.703922 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:42.704166 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:42.704169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:42.704185 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:42.704303 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.704309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.705389 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.705393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.706432 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.706440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.707145 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.707148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.707232 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.707264 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:37:42.707838 (+ 574us) client_negotiation.cc:240] Negotiation successful
0619 18:37:42.707906 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":118,"threads_started":1}
I20260619 18:37:42.708086 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6849b44d33654463b1337994af1689e5" candidate_uuid: "7f472c33a0bf47ac86bf54c950f64f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" is_pre_election: true
I20260619 18:37:42.708240 2295 raft_consensus.cc:2454] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f472c33a0bf47ac86bf54c950f64f91 in term 0.
I20260619 18:37:42.708617 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6849b44d33654463b1337994af1689e5" candidate_uuid: "7f472c33a0bf47ac86bf54c950f64f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" is_pre_election: true
I20260619 18:37:42.708632 2368 leader_election.cc:302] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f8a2e3a2e574a5f8096731a9c40efaf, 7f472c33a0bf47ac86bf54c950f64f91; no voters:
I20260619 18:37:42.708856 2576 raft_consensus.cc:2454] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f472c33a0bf47ac86bf54c950f64f91 in term 0.
I20260619 18:37:42.709088 2652 raft_consensus.cc:2781] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:37:42.709143 2652 raft_consensus.cc:482] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:37:42.709179 2652 raft_consensus.cc:3037] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:42.710239 2652 raft_consensus.cc:504] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.710567 2652 leader_election.cc:288] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f8a2e3a2e574a5f8096731a9c40efaf (127.0.98.1:37819), 8a2bfe2f746641b6b790a7601cdd66f5 (127.0.98.3:43959)
I20260619 18:37:42.710949 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6849b44d33654463b1337994af1689e5" candidate_uuid: "7f472c33a0bf47ac86bf54c950f64f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf"
I20260619 18:37:42.711053 2295 raft_consensus.cc:3037] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:42.711915 2295 raft_consensus.cc:2454] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f472c33a0bf47ac86bf54c950f64f91 in term 1.
I20260619 18:37:42.712136 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6849b44d33654463b1337994af1689e5" candidate_uuid: "7f472c33a0bf47ac86bf54c950f64f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2bfe2f746641b6b790a7601cdd66f5"
I20260619 18:37:42.712188 2368 leader_election.cc:302] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f8a2e3a2e574a5f8096731a9c40efaf, 7f472c33a0bf47ac86bf54c950f64f91; no voters:
I20260619 18:37:42.712260 2576 raft_consensus.cc:3037] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:42.712354 2652 raft_consensus.cc:2781] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:42.712575 2652 raft_consensus.cc:686] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 LEADER]: Becoming Leader. State: Replica: 7f472c33a0bf47ac86bf54c950f64f91, State: Running, Role: LEADER
I20260619 18:37:42.712836 2652 consensus_queue.cc:237] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:42.713275 2576 raft_consensus.cc:2454] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f472c33a0bf47ac86bf54c950f64f91 in term 1.
I20260619 18:37:42.715263 2137 catalog_manager.cc:5495] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f472c33a0bf47ac86bf54c950f64f91 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7f472c33a0bf47ac86bf54c950f64f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } health_report { overall_health: UNKNOWN } } }
I20260619 18:37:42.739025 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.735004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.0.98.2:37763)
0619 18:37:42.735128 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:42.735133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:42.735211 (+ 78us) server_negotiation.cc:407] Connection header received
0619 18:37:42.735376 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:42.735380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:42.735443 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:42.735551 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:37:42.735747 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.736581 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.737602 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.737840 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.737951 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.738201 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:37:42.738650 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:37:42.738765 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:42.738832 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 18:37:42.738888 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:37:42.750439 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.746521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39593 (local address 127.0.98.3:43959)
0619 18:37:42.746640 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:42.746644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:42.746742 (+ 98us) server_negotiation.cc:407] Connection header received
0619 18:37:42.746860 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:42.746863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:42.746914 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:42.746991 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:37:42.747515 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.748286 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.749119 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.749484 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.749601 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.749864 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:37:42.749972 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:37:42.750041 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:42.750230 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 18:37:42.750290 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:37:42.750499 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.746419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43959 (local address 127.0.98.2:39593)
0619 18:37:42.746673 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:42.746685 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:42.746774 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:42.747000 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:42.747001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:42.747010 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:37:42.747419 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.747424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.748398 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.748401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.749010 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.749019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.749604 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.749608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.749695 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.749749 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:37:42.750117 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:37:42.750124 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 18:37:42.750218 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 18:37:42.750342 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":83,"threads_started":1}
I20260619 18:37:42.756233 2295 raft_consensus.cc:1261] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 1 FOLLOWER]: Refusing update from remote peer 7f472c33a0bf47ac86bf54c950f64f91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:42.756341 2576 raft_consensus.cc:1261] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 FOLLOWER]: Refusing update from remote peer 7f472c33a0bf47ac86bf54c950f64f91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:42.756753 2652 consensus_queue.cc:1035] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:42.756963 2694 consensus_queue.cc:1035] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:42.778393 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.774600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37763 (local address 127.0.98.3:44139)
0619 18:37:42.774707 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:42.774722 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:42.774809 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:42.775091 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:42.775094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:42.775101 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:37:42.775201 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.775207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.776204 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.776207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.777071 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:42.777082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.777702 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.777705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:42.777789 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.777818 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:37:42.778108 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:37:42.778114 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:37:42.778167 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 18:37:42.778259 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 18:37:42.778781 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:42.774779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44139 (local address 127.0.98.2:37763)
0619 18:37:42.774886 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:42.774891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:42.774905 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:37:42.774944 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:42.774947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:42.774994 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:42.775072 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:37:42.775303 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.776082 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.777196 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:42.777545 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:42.777631 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:42.777943 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:37:42.778021 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:37:42.778073 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:42.778624 (+ 551us) server_negotiation.cc:299] Negotiation successful
0619 18:37:42.778672 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 18:37:42.779352 2672 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 18:37:44.340994 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:37:44.341146 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:37:44.343528 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa: Sending DeleteTablet for 3 replicas of tablet 6849b44d33654463b1337994af1689e5
I20260619 18:37:44.344046 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet 6849b44d33654463b1337994af1689e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58336
I20260619 18:37:44.344053 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet 6849b44d33654463b1337994af1689e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54802
I20260619 18:37:44.344113 2556 tablet_service.cc:1514] Processing DeleteTablet for tablet 6849b44d33654463b1337994af1689e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59768
I20260619 18:37:44.344332 2727 tablet_replica.cc:330] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91: stopping tablet replica
I20260619 18:37:44.344481 2729 tablet_replica.cc:330] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5: stopping tablet replica
I20260619 18:37:44.344482 2727 raft_consensus.cc:2229] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:37:44.344590 2729 raft_consensus.cc:2229] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:37:44.344774 2727 raft_consensus.cc:2258] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:44.344774 2729 raft_consensus.cc:2258] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:44.345356 2729 ts_tablet_manager.cc:1905] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:44.345356 2727 ts_tablet_manager.cc:1905] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:44.347908 2729 ts_tablet_manager.cc:1918] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:37:44.347908 2727 ts_tablet_manager.cc:1918] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:37:44.348035 2727 log.cc:1196] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/6849b44d33654463b1337994af1689e5
I20260619 18:37:44.348037 2729 log.cc:1196] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/6849b44d33654463b1337994af1689e5
I20260619 18:37:44.348275 2729 ts_tablet_manager.cc:1939] T 6849b44d33654463b1337994af1689e5 P 8a2bfe2f746641b6b790a7601cdd66f5: Deleting consensus metadata
I20260619 18:37:44.348536 2728 tablet_replica.cc:330] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf: stopping tablet replica
I20260619 18:37:44.348661 2728 raft_consensus.cc:2229] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:37:44.348764 2728 raft_consensus.cc:2258] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:44.349476 2728 ts_tablet_manager.cc:1905] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:44.349530 2124 catalog_manager.cc:4841] TS 8a2bfe2f746641b6b790a7601cdd66f5 (127.0.98.3:43959): tablet 6849b44d33654463b1337994af1689e5 (table test-table [id=9535546bf2404b29bb1382cd71b9ef33]) successfully deleted
I20260619 18:37:44.350157 2727 ts_tablet_manager.cc:1939] T 6849b44d33654463b1337994af1689e5 P 7f472c33a0bf47ac86bf54c950f64f91: Deleting consensus metadata
I20260619 18:37:44.350848 2126 catalog_manager.cc:4841] TS 7f472c33a0bf47ac86bf54c950f64f91 (127.0.98.2:37763): tablet 6849b44d33654463b1337994af1689e5 (table test-table [id=9535546bf2404b29bb1382cd71b9ef33]) successfully deleted
I20260619 18:37:44.352432 2728 ts_tablet_manager.cc:1918] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:37:44.352504 2728 log.cc:1196] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/wal/wals/6849b44d33654463b1337994af1689e5
I20260619 18:37:44.352773 2728 ts_tablet_manager.cc:1939] T 6849b44d33654463b1337994af1689e5 P 1f8a2e3a2e574a5f8096731a9c40efaf: Deleting consensus metadata
I20260619 18:37:44.353576 2124 catalog_manager.cc:4841] TS 1f8a2e3a2e574a5f8096731a9c40efaf (127.0.98.1:37819): tablet 6849b44d33654463b1337994af1689e5 (table test-table [id=9535546bf2404b29bb1382cd71b9ef33]) successfully deleted
Jun 19 18:37:44 dist-test-slave-rxv6 krb5kdc[2094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894264, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:37:44 dist-test-slave-rxv6 krb5kdc[2094](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894264, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:44.377585 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:44.367317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60610 (local address 127.0.98.62:41049)
0619 18:37:44.367588 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:44.367595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:44.367609 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:37:44.367651 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:44.367654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:44.367696 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:44.367781 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:44.368710 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.369426 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.370636 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.370971 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.371053 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:44.373807 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:44.373810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:44.373812 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:44.373836 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:44.375534 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:44.375981 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:44.375985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:44.375987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:44.376087 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:44.376370 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:44.376373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:44.376375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:44.376607 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:44.376733 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:44.377276 (+ 543us) server_negotiation.cc:299] Negotiation successful
0619 18:37:44.377404 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
I20260619 18:37:44.380307 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:37:44.380817 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:37:44.385653 2276 tablet_service.cc:1467] Processing CreateTablet for tablet 07c1f8ad20e5407f911e48e4d87f6ea5 (DEFAULT_TABLE table=test-table [id=e48689c35e004faa82f617c8ccf9ae9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:44.385697 2416 tablet_service.cc:1467] Processing CreateTablet for tablet 07c1f8ad20e5407f911e48e4d87f6ea5 (DEFAULT_TABLE table=test-table [id=e48689c35e004faa82f617c8ccf9ae9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:44.385991 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c1f8ad20e5407f911e48e4d87f6ea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:44.386003 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c1f8ad20e5407f911e48e4d87f6ea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:44.386154 2556 tablet_service.cc:1467] Processing CreateTablet for tablet 07c1f8ad20e5407f911e48e4d87f6ea5 (DEFAULT_TABLE table=test-table [id=e48689c35e004faa82f617c8ccf9ae9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:44.386399 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c1f8ad20e5407f911e48e4d87f6ea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:44.388064 2738 tablet_bootstrap.cc:492] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91: Bootstrap starting.
I20260619 18:37:44.388670 2740 tablet_bootstrap.cc:492] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5: Bootstrap starting.
I20260619 18:37:44.388842 2739 tablet_bootstrap.cc:492] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf: Bootstrap starting.
I20260619 18:37:44.389163 2738 tablet_bootstrap.cc:654] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:44.389611 2740 tablet_bootstrap.cc:654] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:44.390220 2739 tablet_bootstrap.cc:654] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:44.390405 2738 tablet_bootstrap.cc:492] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91: No bootstrap required, opened a new log
I20260619 18:37:44.390534 2738 ts_tablet_manager.cc:1397] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:37:44.390686 2740 tablet_bootstrap.cc:492] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5: No bootstrap required, opened a new log
I20260619 18:37:44.390786 2740 ts_tablet_manager.cc:1397] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:37:44.391111 2738 raft_consensus.cc:348] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.391197 2740 raft_consensus.cc:348] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.391312 2740 raft_consensus.cc:374] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:44.391336 2739 tablet_bootstrap.cc:492] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf: No bootstrap required, opened a new log
I20260619 18:37:44.391352 2740 raft_consensus.cc:729] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2bfe2f746641b6b790a7601cdd66f5, State: Initialized, Role: FOLLOWER
I20260619 18:37:44.391403 2739 ts_tablet_manager.cc:1397] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:37:44.391294 2738 raft_consensus.cc:374] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:44.391469 2740 consensus_queue.cc:260] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.391597 2738 raft_consensus.cc:729] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f472c33a0bf47ac86bf54c950f64f91, State: Initialized, Role: FOLLOWER
I20260619 18:37:44.391738 2740 ts_tablet_manager.cc:1428] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:37:44.391800 2739 raft_consensus.cc:348] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.391904 2739 raft_consensus.cc:374] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:44.391765 2738 consensus_queue.cc:260] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.391942 2739 raft_consensus.cc:729] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f8a2e3a2e574a5f8096731a9c40efaf, State: Initialized, Role: FOLLOWER
I20260619 18:37:44.392051 2739 consensus_queue.cc:260] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.392277 2738 ts_tablet_manager.cc:1428] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:37:44.392398 2739 ts_tablet_manager.cc:1428] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:37:44.488579 2718 raft_consensus.cc:482] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:37:44.488727 2718 raft_consensus.cc:504] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.489205 2718 leader_election.cc:288] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f8a2e3a2e574a5f8096731a9c40efaf (127.0.98.1:37819), 8a2bfe2f746641b6b790a7601cdd66f5 (127.0.98.3:43959)
I20260619 18:37:44.489771 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c1f8ad20e5407f911e48e4d87f6ea5" candidate_uuid: "7f472c33a0bf47ac86bf54c950f64f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" is_pre_election: true
I20260619 18:37:44.489771 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c1f8ad20e5407f911e48e4d87f6ea5" candidate_uuid: "7f472c33a0bf47ac86bf54c950f64f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" is_pre_election: true
I20260619 18:37:44.489948 2576 raft_consensus.cc:2454] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f472c33a0bf47ac86bf54c950f64f91 in term 0.
I20260619 18:37:44.489948 2296 raft_consensus.cc:2454] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f472c33a0bf47ac86bf54c950f64f91 in term 0.
I20260619 18:37:44.490280 2368 leader_election.cc:302] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f472c33a0bf47ac86bf54c950f64f91, 8a2bfe2f746641b6b790a7601cdd66f5; no voters:
I20260619 18:37:44.490489 2718 raft_consensus.cc:2781] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:37:44.490557 2718 raft_consensus.cc:482] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:37:44.490599 2718 raft_consensus.cc:3037] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:44.491564 2718 raft_consensus.cc:504] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.491879 2718 leader_election.cc:288] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [CANDIDATE]: Term 1 election: Requested vote from peers 1f8a2e3a2e574a5f8096731a9c40efaf (127.0.98.1:37819), 8a2bfe2f746641b6b790a7601cdd66f5 (127.0.98.3:43959)
I20260619 18:37:44.492271 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c1f8ad20e5407f911e48e4d87f6ea5" candidate_uuid: "7f472c33a0bf47ac86bf54c950f64f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2bfe2f746641b6b790a7601cdd66f5"
I20260619 18:37:44.492312 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c1f8ad20e5407f911e48e4d87f6ea5" candidate_uuid: "7f472c33a0bf47ac86bf54c950f64f91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf"
I20260619 18:37:44.492429 2576 raft_consensus.cc:3037] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:44.492481 2296 raft_consensus.cc:3037] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:44.493407 2576 raft_consensus.cc:2454] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f472c33a0bf47ac86bf54c950f64f91 in term 1.
I20260619 18:37:44.493399 2296 raft_consensus.cc:2454] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f472c33a0bf47ac86bf54c950f64f91 in term 1.
I20260619 18:37:44.493721 2368 leader_election.cc:302] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f472c33a0bf47ac86bf54c950f64f91, 8a2bfe2f746641b6b790a7601cdd66f5; no voters:
I20260619 18:37:44.493896 2718 raft_consensus.cc:2781] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:44.494019 2718 raft_consensus.cc:686] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 LEADER]: Becoming Leader. State: Replica: 7f472c33a0bf47ac86bf54c950f64f91, State: Running, Role: LEADER
I20260619 18:37:44.494159 2718 consensus_queue.cc:237] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.495779 2138 catalog_manager.cc:5495] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f472c33a0bf47ac86bf54c950f64f91 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7f472c33a0bf47ac86bf54c950f64f91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } health_report { overall_health: UNKNOWN } } }
I20260619 18:37:44.520027 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:44.516858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.0.98.2:37763)
0619 18:37:44.517166 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:44.517172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:44.517188 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:37:44.517268 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:44.517270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:44.517314 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:44.517388 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:37:44.517658 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.518195 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.519069 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.519305 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.519380 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:44.519643 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:37:44.519714 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:37:44.519760 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:44.519832 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 18:37:44.519874 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":139,"threads_started":1}
I20260619 18:37:44.526799 2576 raft_consensus.cc:1261] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 FOLLOWER]: Refusing update from remote peer 7f472c33a0bf47ac86bf54c950f64f91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:44.526823 2296 raft_consensus.cc:1261] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 1 FOLLOWER]: Refusing update from remote peer 7f472c33a0bf47ac86bf54c950f64f91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:44.527202 2704 consensus_queue.cc:1035] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:44.527349 2718 consensus_queue.cc:1035] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:44.557665 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:37:44.557811 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:37:44.559692 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa: Sending DeleteTablet for 3 replicas of tablet 07c1f8ad20e5407f911e48e4d87f6ea5
I20260619 18:37:44.560169 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c1f8ad20e5407f911e48e4d87f6ea5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54802
I20260619 18:37:44.560173 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c1f8ad20e5407f911e48e4d87f6ea5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58336
I20260619 18:37:44.560293 2556 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c1f8ad20e5407f911e48e4d87f6ea5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59768
I20260619 18:37:44.560350 2728 tablet_replica.cc:330] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf: stopping tablet replica
I20260619 18:37:44.560467 2728 raft_consensus.cc:2229] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:37:44.560530 2729 tablet_replica.cc:330] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5: stopping tablet replica
I20260619 18:37:44.560582 2728 raft_consensus.cc:2258] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:44.560650 2729 raft_consensus.cc:2229] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:37:44.560753 2727 tablet_replica.cc:330] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91: stopping tablet replica
I20260619 18:37:44.560760 2729 raft_consensus.cc:2258] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:44.560863 2727 raft_consensus.cc:2229] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:37:44.561028 2727 raft_consensus.cc:2258] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:44.561221 2138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 18:37:44.561384 2729 ts_tablet_manager.cc:1905] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 18:37:44.561600 2138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:37:44.561614 2727 ts_tablet_manager.cc:1905] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:44.561646 2728 ts_tablet_manager.cc:1905] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:44.563915 2728 ts_tablet_manager.cc:1918] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:37:44.563915 2727 ts_tablet_manager.cc:1918] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:37:44.563980 2728 log.cc:1196] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-0/wal/wals/07c1f8ad20e5407f911e48e4d87f6ea5
I20260619 18:37:44.563983 2727 log.cc:1196] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/07c1f8ad20e5407f911e48e4d87f6ea5
I20260619 18:37:44.564033 2729 ts_tablet_manager.cc:1918] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:37:44.564100 2729 log.cc:1196] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/07c1f8ad20e5407f911e48e4d87f6ea5
I20260619 18:37:44.564210 2727 ts_tablet_manager.cc:1939] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 7f472c33a0bf47ac86bf54c950f64f91: Deleting consensus metadata
I20260619 18:37:44.564209 2728 ts_tablet_manager.cc:1939] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 1f8a2e3a2e574a5f8096731a9c40efaf: Deleting consensus metadata
I20260619 18:37:44.564383 2729 ts_tablet_manager.cc:1939] T 07c1f8ad20e5407f911e48e4d87f6ea5 P 8a2bfe2f746641b6b790a7601cdd66f5: Deleting consensus metadata
I20260619 18:37:44.564962 2124 catalog_manager.cc:4841] TS 8a2bfe2f746641b6b790a7601cdd66f5 (127.0.98.3:43959): tablet 07c1f8ad20e5407f911e48e4d87f6ea5 (table test-table [id=e48689c35e004faa82f617c8ccf9ae9b]) successfully deleted
I20260619 18:37:44.565117 2124 catalog_manager.cc:4841] TS 1f8a2e3a2e574a5f8096731a9c40efaf (127.0.98.1:37819): tablet 07c1f8ad20e5407f911e48e4d87f6ea5 (table test-table [id=e48689c35e004faa82f617c8ccf9ae9b]) successfully deleted
I20260619 18:37:44.565462 2126 catalog_manager.cc:4841] TS 7f472c33a0bf47ac86bf54c950f64f91 (127.0.98.2:37763): tablet 07c1f8ad20e5407f911e48e4d87f6ea5 (table test-table [id=e48689c35e004faa82f617c8ccf9ae9b]) successfully deleted
I20260619 18:37:44.566009 2276 tablet_service.cc:1467] Processing CreateTablet for tablet bf92cf311fb5462e9fc75bd5be6ec7d1 (DEFAULT_TABLE table=test-table [id=16ff6f3b07cf4deeab54e49b61908324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:44.566061 2416 tablet_service.cc:1467] Processing CreateTablet for tablet bf92cf311fb5462e9fc75bd5be6ec7d1 (DEFAULT_TABLE table=test-table [id=16ff6f3b07cf4deeab54e49b61908324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:44.566249 2276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf92cf311fb5462e9fc75bd5be6ec7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:44.566249 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf92cf311fb5462e9fc75bd5be6ec7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:44.566254 2556 tablet_service.cc:1467] Processing CreateTablet for tablet bf92cf311fb5462e9fc75bd5be6ec7d1 (DEFAULT_TABLE table=test-table [id=16ff6f3b07cf4deeab54e49b61908324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:44.566465 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf92cf311fb5462e9fc75bd5be6ec7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:44.568199 2740 tablet_bootstrap.cc:492] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5: Bootstrap starting.
I20260619 18:37:44.568310 2738 tablet_bootstrap.cc:492] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91: Bootstrap starting.
I20260619 18:37:44.568992 2740 tablet_bootstrap.cc:654] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:44.569013 2738 tablet_bootstrap.cc:654] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:44.569418 2739 tablet_bootstrap.cc:492] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf: Bootstrap starting.
I20260619 18:37:44.569929 2738 tablet_bootstrap.cc:492] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91: No bootstrap required, opened a new log
I20260619 18:37:44.570024 2738 ts_tablet_manager.cc:1397] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:37:44.570091 2740 tablet_bootstrap.cc:492] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5: No bootstrap required, opened a new log
I20260619 18:37:44.570173 2740 ts_tablet_manager.cc:1397] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:37:44.570286 2739 tablet_bootstrap.cc:654] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:44.570438 2738 raft_consensus.cc:348] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.570533 2738 raft_consensus.cc:374] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:44.570567 2738 raft_consensus.cc:729] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f472c33a0bf47ac86bf54c950f64f91, State: Initialized, Role: FOLLOWER
I20260619 18:37:44.570545 2740 raft_consensus.cc:348] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.570654 2740 raft_consensus.cc:374] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:44.570684 2740 raft_consensus.cc:729] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2bfe2f746641b6b790a7601cdd66f5, State: Initialized, Role: FOLLOWER
I20260619 18:37:44.570679 2738 consensus_queue.cc:260] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.570801 2740 consensus_queue.cc:260] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.570930 2738 ts_tablet_manager.cc:1428] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:37:44.571060 2740 ts_tablet_manager.cc:1428] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:37:44.571470 2739 tablet_bootstrap.cc:492] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf: No bootstrap required, opened a new log
I20260619 18:37:44.571563 2739 ts_tablet_manager.cc:1397] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:37:44.571936 2739 raft_consensus.cc:348] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.572039 2739 raft_consensus.cc:374] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:44.572067 2739 raft_consensus.cc:729] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f8a2e3a2e574a5f8096731a9c40efaf, State: Initialized, Role: FOLLOWER
I20260619 18:37:44.572175 2739 consensus_queue.cc:260] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.572417 2739 ts_tablet_manager.cc:1428] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 18:37:44.642174 2744 raft_consensus.cc:482] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:37:44.642333 2744 raft_consensus.cc:504] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.643404 2744 leader_election.cc:288] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f472c33a0bf47ac86bf54c950f64f91 (127.0.98.2:37763), 8a2bfe2f746641b6b790a7601cdd66f5 (127.0.98.3:43959)
I20260619 18:37:44.647558 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:44.643625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37763 (local address 127.0.98.1:58881)
0619 18:37:44.643908 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:44.643921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:44.644044 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:44.644302 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:44.644305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:44.644321 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:44.644464 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:44.644470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.645299 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.645301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:44.646354 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:44.646364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.647196 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.647199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:44.647277 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:44.647297 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 18:37:44.647355 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 18:37:44.647408 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":4,"thread_start_us":71,"threads_started":1}
I20260619 18:37:44.647588 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:44.643704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58881 (local address 127.0.98.2:37763)
0619 18:37:44.643912 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:44.643916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:44.643998 (+ 82us) server_negotiation.cc:407] Connection header received
0619 18:37:44.644171 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:44.644174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:44.644221 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:44.644281 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:44.644567 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.645153 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.646504 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.647102 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.647197 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:44.647258 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:44.647383 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 18:37:44.647462 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260619 18:37:44.647722 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:44.643681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43959 (local address 127.0.98.1:45971)
0619 18:37:44.644008 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:44.644022 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:44.644120 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:44.644357 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:44.644360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:44.644395 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:44.644540 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:44.644545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.645241 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.645244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:44.646331 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:44.646341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.647330 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.647334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:44.647427 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:44.647448 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:37:44.647531 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 18:37:44.647594 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":26,"thread_start_us":104,"threads_started":1}
I20260619 18:37:44.647898 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:44.643758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45971 (local address 127.0.98.3:43959)
0619 18:37:44.644069 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:44.644083 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:44.644098 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:37:44.644221 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:44.644223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:44.644262 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:44.644330 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:44.644638 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.645142 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.646457 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.647088 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.647197 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:44.647258 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:44.647560 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 18:37:44.647764 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260619 18:37:44.647979 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf92cf311fb5462e9fc75bd5be6ec7d1" candidate_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f472c33a0bf47ac86bf54c950f64f91" is_pre_election: true
I20260619 18:37:44.648137 2435 raft_consensus.cc:2454] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f8a2e3a2e574a5f8096731a9c40efaf in term 0.
I20260619 18:37:44.648260 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf92cf311fb5462e9fc75bd5be6ec7d1" candidate_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" is_pre_election: true
I20260619 18:37:44.648387 2576 raft_consensus.cc:2454] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f8a2e3a2e574a5f8096731a9c40efaf in term 0.
I20260619 18:37:44.648550 2230 leader_election.cc:302] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f8a2e3a2e574a5f8096731a9c40efaf, 7f472c33a0bf47ac86bf54c950f64f91; no voters:
I20260619 18:37:44.648854 2744 raft_consensus.cc:2781] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:37:44.648908 2744 raft_consensus.cc:482] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:37:44.648954 2744 raft_consensus.cc:3037] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:44.650017 2744 raft_consensus.cc:504] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.650343 2744 leader_election.cc:288] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [CANDIDATE]: Term 1 election: Requested vote from peers 7f472c33a0bf47ac86bf54c950f64f91 (127.0.98.2:37763), 8a2bfe2f746641b6b790a7601cdd66f5 (127.0.98.3:43959)
I20260619 18:37:44.650666 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf92cf311fb5462e9fc75bd5be6ec7d1" candidate_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a2bfe2f746641b6b790a7601cdd66f5"
I20260619 18:37:44.650776 2576 raft_consensus.cc:3037] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:44.650736 2435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf92cf311fb5462e9fc75bd5be6ec7d1" candidate_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f472c33a0bf47ac86bf54c950f64f91"
I20260619 18:37:44.650882 2435 raft_consensus.cc:3037] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:44.651686 2576 raft_consensus.cc:2454] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f8a2e3a2e574a5f8096731a9c40efaf in term 1.
I20260619 18:37:44.651798 2435 raft_consensus.cc:2454] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f8a2e3a2e574a5f8096731a9c40efaf in term 1.
I20260619 18:37:44.651959 2228 leader_election.cc:302] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f8a2e3a2e574a5f8096731a9c40efaf, 8a2bfe2f746641b6b790a7601cdd66f5; no voters:
I20260619 18:37:44.652128 2744 raft_consensus.cc:2781] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:44.652320 2744 raft_consensus.cc:686] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [term 1 LEADER]: Becoming Leader. State: Replica: 1f8a2e3a2e574a5f8096731a9c40efaf, State: Running, Role: LEADER
I20260619 18:37:44.652576 2744 consensus_queue.cc:237] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } }
I20260619 18:37:44.654855 2139 catalog_manager.cc:5495] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f8a2e3a2e574a5f8096731a9c40efaf (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f8a2e3a2e574a5f8096731a9c40efaf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 } health_report { overall_health: UNKNOWN } } }
I20260619 18:37:44.678651 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:44.674588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.0.98.1:37819)
0619 18:37:44.674882 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:44.674888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:44.674898 (+ 10us) server_negotiation.cc:407] Connection header received
0619 18:37:44.674966 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:44.674968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:44.675025 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:44.675141 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:37:44.675361 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.675892 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.676674 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.677408 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.677511 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:44.677746 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:37:44.678255 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:37:44.678360 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:44.678436 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 18:37:44.678493 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":125,"threads_started":1}
I20260619 18:37:44.688252 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:44.685589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44847 (local address 127.0.98.3:43959)
0619 18:37:44.685693 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:44.685697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:44.685711 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:37:44.685808 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:44.685810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:44.685854 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:44.685953 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:37:44.686146 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.686629 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.687404 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.687622 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.687696 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:44.687923 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:37:44.687995 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:37:44.688043 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:44.688104 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 18:37:44.688147 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 18:37:44.688280 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:44.685346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43959 (local address 127.0.98.1:44847)
0619 18:37:44.685645 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:44.685658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:44.685736 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:44.685961 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:44.685963 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:44.685969 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:37:44.686066 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:44.686070 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.686701 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.686703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:44.687311 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:44.687318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.687709 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.687712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:44.687779 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:44.687819 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:37:44.688050 (+ 231us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:37:44.688054 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 18:37:44.688091 (+ 37us) client_negotiation.cc:240] Negotiation successful
0619 18:37:44.688151 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
I20260619 18:37:44.693763 2435 raft_consensus.cc:1261] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 FOLLOWER]: Refusing update from remote peer 1f8a2e3a2e574a5f8096731a9c40efaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:44.693825 2576 raft_consensus.cc:1261] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 FOLLOWER]: Refusing update from remote peer 1f8a2e3a2e574a5f8096731a9c40efaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:44.694208 2758 consensus_queue.cc:1035] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f472c33a0bf47ac86bf54c950f64f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:44.694375 2744 consensus_queue.cc:1035] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 1f8a2e3a2e574a5f8096731a9c40efaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a2bfe2f746641b6b790a7601cdd66f5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:44.710719 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:44.707246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45375 (local address 127.0.98.1:37819)
0619 18:37:44.707347 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:44.707352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:44.707718 (+ 366us) server_negotiation.cc:407] Connection header received
0619 18:37:44.707823 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:44.707826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:44.707877 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:44.707948 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:37:44.708149 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.708665 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.709586 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.709886 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.710015 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:44.710278 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:37:44.710361 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:37:44.710429 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:44.710534 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 18:37:44.710603 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 18:37:44.710767 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:44.707194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37819 (local address 127.0.98.3:45375)
0619 18:37:44.707625 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:44.707638 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:44.707742 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:44.707958 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:44.707961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:44.707969 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:37:44.708077 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:44.708083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.708745 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.708747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:44.709459 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:44.709470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:44.710063 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:44.710067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:44.710149 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:44.710174 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:37:44.710445 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:37:44.710451 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:37:44.710508 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 18:37:44.710621 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":95,"threads_started":1}
W20260619 18:37:44.711526 2710 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 18:37:46.270273 2138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:37:46.270507 2138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:37:46.273437 2138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d98dd1fdde74542ba4376f8ab70d2fa: Sending DeleteTablet for 3 replicas of tablet bf92cf311fb5462e9fc75bd5be6ec7d1
I20260619 18:37:46.274134 2276 tablet_service.cc:1514] Processing DeleteTablet for tablet bf92cf311fb5462e9fc75bd5be6ec7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54802
I20260619 18:37:46.274171 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet bf92cf311fb5462e9fc75bd5be6ec7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58336
I20260619 18:37:46.274215 2556 tablet_service.cc:1514] Processing DeleteTablet for tablet bf92cf311fb5462e9fc75bd5be6ec7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59768
I20260619 18:37:46.274573 2793 tablet_replica.cc:330] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5: stopping tablet replica
I20260619 18:37:46.274585 2792 tablet_replica.cc:330] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91: stopping tablet replica
I20260619 18:37:46.274700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 2211
I20260619 18:37:46.274735 2792 raft_consensus.cc:2229] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:37:46.275833 2792 raft_consensus.cc:2258] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:46.276410 2793 raft_consensus.cc:2229] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:37:46.276593 2793 raft_consensus.cc:2258] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:46.276645 2792 ts_tablet_manager.cc:1905] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:46.277334 2793 ts_tablet_manager.cc:1905] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:46.279964 2792 ts_tablet_manager.cc:1918] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:37:46.280061 2792 log.cc:1196] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/bf92cf311fb5462e9fc75bd5be6ec7d1
I20260619 18:37:46.280279 2793 ts_tablet_manager.cc:1918] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 18:37:46.280355 2792 ts_tablet_manager.cc:1939] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 7f472c33a0bf47ac86bf54c950f64f91: Deleting consensus metadata
I20260619 18:37:46.280355 2793 log.cc:1196] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/bf92cf311fb5462e9fc75bd5be6ec7d1
I20260619 18:37:46.280591 2793 ts_tablet_manager.cc:1939] T bf92cf311fb5462e9fc75bd5be6ec7d1 P 8a2bfe2f746641b6b790a7601cdd66f5: Deleting consensus metadata
I20260619 18:37:46.281327 2124 catalog_manager.cc:4841] TS 8a2bfe2f746641b6b790a7601cdd66f5 (127.0.98.3:43959): tablet bf92cf311fb5462e9fc75bd5be6ec7d1 (table test-table [id=16ff6f3b07cf4deeab54e49b61908324]) successfully deleted
I20260619 18:37:46.281319 2126 catalog_manager.cc:4841] TS 7f472c33a0bf47ac86bf54c950f64f91 (127.0.98.2:37763): tablet bf92cf311fb5462e9fc75bd5be6ec7d1 (table test-table [id=16ff6f3b07cf4deeab54e49b61908324]) successfully deleted
W20260619 18:37:46.282663 2124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37819 (error 108)
I20260619 18:37:46.282939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 2351
I20260619 18:37:46.285159 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:46.284649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37819 (local address 127.0.0.1:40344)
0619 18:37:46.284941 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:46.285032 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":100,"threads_started":1}
W20260619 18:37:46.285326 2124 catalog_manager.cc:4584] TS 1f8a2e3a2e574a5f8096731a9c40efaf (127.0.98.1:37819): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf92cf311fb5462e9fc75bd5be6ec7d1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37819: connect: Connection refused (error 111)
I20260619 18:37:46.289709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 2491
I20260619 18:37:46.296902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 2107
2026-06-19T18:37:46Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6080 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:37:46 dist-test-slave-rxv6 krb5kdc[2798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:37:46 dist-test-slave-rxv6 krb5kdc[2798](info): set up 2 sockets
Jun 19 18:37:46 dist-test-slave-rxv6 krb5kdc[2798](info): commencing operation
krb5kdc: starting...
W20260619 18:37:47.273775 392 mini_kdc.cc:121] Time spent starting KDC: real 0.953s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:37:47 dist-test-slave-rxv6 krb5kdc[2798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:37:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:37:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:37:47.388432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43537
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:47.497228 2811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:47.497501 2811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:47.497540 2811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:47.501510 2811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:37:47.501578 2811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:47.501596 2811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:47.501612 2811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:37:47.501626 2811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:37:47.507004 2811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:47.507984 2811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:47.508945 2811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:47.517230 2817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:47.517244 2816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:47.517248 2819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:47.517314 2811 server_base.cc:1034] running on GCE node
I20260619 18:37:47.518069 2811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:47.519167 2811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:47.520328 2811 hybrid_clock.cc:648] HybridClock initialized: now 1781894267520315 us; error 24 us; skew 500 ppm
Jun 19 18:37:47 dist-test-slave-rxv6 krb5kdc[2798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:47.523003 2811 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:37:47.523820 2811 webserver.cc:460] Webserver started at http://127.0.98.62:44181/ using document root <none> and password file <none>
I20260619 18:37:47.524339 2811 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:47.524401 2811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:47.524580 2811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:47.526585 2811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "ce5a5d2361554184b6a0b90628d087bb"
format_stamp: "Formatted at 2026-06-19 18:37:47 on dist-test-slave-rxv6"
server_key: "1fc45d90a8d14437e6e1af75f57a6672"
server_key_iv: "71c7eaf5ab965083357331a913db27d3"
server_key_version: "encryptionkey@0"
I20260619 18:37:47.527103 2811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "ce5a5d2361554184b6a0b90628d087bb"
format_stamp: "Formatted at 2026-06-19 18:37:47 on dist-test-slave-rxv6"
server_key: "1fc45d90a8d14437e6e1af75f57a6672"
server_key_iv: "71c7eaf5ab965083357331a913db27d3"
server_key_version: "encryptionkey@0"
I20260619 18:37:47.530634 2811 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:37:47.532953 2826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:47.534164 2811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:37:47.534274 2811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "ce5a5d2361554184b6a0b90628d087bb"
format_stamp: "Formatted at 2026-06-19 18:37:47 on dist-test-slave-rxv6"
server_key: "1fc45d90a8d14437e6e1af75f57a6672"
server_key_iv: "71c7eaf5ab965083357331a913db27d3"
server_key_version: "encryptionkey@0"
I20260619 18:37:47.534365 2811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:47.554575 2811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:47.557533 2811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:47.557715 2811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:47.564236 2811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43537
I20260619 18:37:47.564270 2878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43537 every 8 connection(s)
I20260619 18:37:47.565162 2811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:37:47.568094 2879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:47.573674 2879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb: Bootstrap starting.
I20260619 18:37:47.574509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 2811
I20260619 18:37:47.574608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:37:47.574848 392 external_mini_cluster.cc:1428] Setting key 35ee77ba82fb6e1dcccb855fdf504c58
I20260619 18:37:47.576563 2879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:47.577446 2879 log.cc:824] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:47.579578 2879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb: No bootstrap required, opened a new log
I20260619 18:37:47.583619 2879 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5a5d2361554184b6a0b90628d087bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43537 } }
I20260619 18:37:47.583878 2879 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:47.583935 2879 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5a5d2361554184b6a0b90628d087bb, State: Initialized, Role: FOLLOWER
Jun 19 18:37:47 dist-test-slave-rxv6 krb5kdc[2798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:47.584391 2879 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5a5d2361554184b6a0b90628d087bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43537 } }
I20260619 18:37:47.584554 2879 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:37:47.584625 2879 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:37:47.584729 2879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:47.585675 2879 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5a5d2361554184b6a0b90628d087bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43537 } }
I20260619 18:37:47.586058 2879 leader_election.cc:302] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce5a5d2361554184b6a0b90628d087bb; no voters:
I20260619 18:37:47.586370 2879 leader_election.cc:288] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:37:47.586493 2884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:47.586648 2884 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [term 1 LEADER]: Becoming Leader. State: Replica: ce5a5d2361554184b6a0b90628d087bb, State: Running, Role: LEADER
I20260619 18:37:47.587009 2884 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5a5d2361554184b6a0b90628d087bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43537 } }
I20260619 18:37:47.587704 2879 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:37:47.588204 2884 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce5a5d2361554184b6a0b90628d087bb. Latest consensus state: current_term: 1 leader_uuid: "ce5a5d2361554184b6a0b90628d087bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5a5d2361554184b6a0b90628d087bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43537 } } }
I20260619 18:37:47.588945 2884 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:47.588832 2885 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce5a5d2361554184b6a0b90628d087bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce5a5d2361554184b6a0b90628d087bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43537 } } }
I20260619 18:37:47.589231 2885 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:47.589567 2893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:37:47.590412 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:47.576192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.0.98.62:43537)
0619 18:37:47.576677 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:47.576697 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:47.576740 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:37:47.577584 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:47.577606 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:47.578053 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:47.578401 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:47.579019 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:47.580491 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:47.581388 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:47.582148 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:47.582355 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:47.584938 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:47.584943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:47.584957 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:47.584987 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:47.586805 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:47.587265 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:47.587271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:47.587276 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:47.587397 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:47.587665 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:47.587668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:47.587670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:47.588120 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:47.588900 (+ 780us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:47.589051 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 18:37:47.589348 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":138,"threads_started":1}
I20260619 18:37:47.592660 2893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:37:47.598003 2893 catalog_manager.cc:1347] Generated new cluster ID: 9b7841c9dfec4443bc9bb41a6d072b32
I20260619 18:37:47.598078 2893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:37:47.617698 2893 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:37:47.618888 2893 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:37:47.625753 2893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce5a5d2361554184b6a0b90628d087bb: Generated new TSK 0
I20260619 18:37:47.626472 2893 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:37:47.656937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43537
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:47.763679 2905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:47.763932 2905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:47.763991 2905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:47.768008 2905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:47.768067 2905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:47.768143 2905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:37:47.773288 2905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:47.774307 2905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:47.775285 2905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:47.784279 2910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:47.784279 2913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:47.784328 2905 server_base.cc:1034] running on GCE node
W20260619 18:37:47.784284 2911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:47.784902 2905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:47.785511 2905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:47.786698 2905 hybrid_clock.cc:648] HybridClock initialized: now 1781894267786680 us; error 28 us; skew 500 ppm
Jun 19 18:37:47 dist-test-slave-rxv6 krb5kdc[2798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:47.789454 2905 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:37:47.790314 2905 webserver.cc:460] Webserver started at http://127.0.98.1:37013/ using document root <none> and password file <none>
I20260619 18:37:47.790815 2905 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:47.790874 2905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:47.791035 2905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:47.793135 2905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "3b50b82d68b3400f80ba8f2ba1797c6f"
format_stamp: "Formatted at 2026-06-19 18:37:47 on dist-test-slave-rxv6"
server_key: "8294735296c34cb2ec3982ad3aab11e4"
server_key_iv: "15a0dd4a15e480fba5bc7c933d663a28"
server_key_version: "encryptionkey@0"
I20260619 18:37:47.793815 2905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3b50b82d68b3400f80ba8f2ba1797c6f"
format_stamp: "Formatted at 2026-06-19 18:37:47 on dist-test-slave-rxv6"
server_key: "8294735296c34cb2ec3982ad3aab11e4"
server_key_iv: "15a0dd4a15e480fba5bc7c933d663a28"
server_key_version: "encryptionkey@0"
I20260619 18:37:47.798251 2905 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:37:47.800601 2920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:47.801806 2905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:37:47.801926 2905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "3b50b82d68b3400f80ba8f2ba1797c6f"
format_stamp: "Formatted at 2026-06-19 18:37:47 on dist-test-slave-rxv6"
server_key: "8294735296c34cb2ec3982ad3aab11e4"
server_key_iv: "15a0dd4a15e480fba5bc7c933d663a28"
server_key_version: "encryptionkey@0"
I20260619 18:37:47.802026 2905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:47.817020 2905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:47.820060 2905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:47.820226 2905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:47.820783 2905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:47.821569 2905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:47.821615 2905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:47.821657 2905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:47.821686 2905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:47.829955 2905 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46813
I20260619 18:37:47.830013 3033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46813 every 8 connection(s)
I20260619 18:37:47.830826 2905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:37:47.832959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 2905
I20260619 18:37:47.833050 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:37:47.833329 392 external_mini_cluster.cc:1428] Setting key a8be5978bce96698c613a88710813bce
Jun 19 18:37:47 dist-test-slave-rxv6 krb5kdc[2798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 18:37:47.844717 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:47.832562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35205 (local address 127.0.98.62:43537)
0619 18:37:47.832712 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:47.832717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:47.833593 (+ 876us) server_negotiation.cc:407] Connection header received
0619 18:37:47.835199 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:47.835202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:47.835243 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:47.835313 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:47.836731 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:47.837327 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:47.838603 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:47.838989 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:47.839100 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:47.841345 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:47.841359 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:47.841363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:47.841392 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:47.842741 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:47.843243 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:47.843246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:47.843247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:47.843323 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:47.843649 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:47.843651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:47.843652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:47.843817 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:47.843886 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:47.844406 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 18:37:47.844511 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:37:47.845480 3036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:47.832947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43537 (local address 127.0.98.1:35205)
0619 18:37:47.833427 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:47.833483 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:47.834961 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:47.835515 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:47.835524 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:47.835991 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:47.836596 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:47.836610 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:47.837500 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:47.837503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:47.838495 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:47.838505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:47.839122 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:47.839124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:47.839255 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:47.839816 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:47.839837 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:47.841175 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:47.842853 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:47.842857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:47.842869 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:47.843150 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:47.843457 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:47.843459 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:47.843460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:47.843584 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:47.843892 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:47.843897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:47.844147 (+ 250us) client_negotiation.cc:769] Sending connection context
0619 18:37:47.844306 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 18:37:47.844650 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":133,"threads_started":1}
I20260619 18:37:47.846756 3034 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43537
I20260619 18:37:47.846980 3034 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:47.847553 3034 heartbeater.cc:511] Master 127.0.98.62:43537 requested a full tablet report, sending...
I20260619 18:37:47.849248 2843 ts_manager.cc:194] Registered new tserver with Master: 3b50b82d68b3400f80ba8f2ba1797c6f (127.0.98.1:46813)
I20260619 18:37:47.850875 2843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:37:47.861090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43537
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:47.968118 3039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:47.968369 3039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:47.968410 3039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:47.972330 3039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:47.972384 3039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:47.972441 3039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:37:47.977609 3039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:47.978711 3039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:47.979684 3039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:47.989171 3044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:47.989179 3047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:47.989310 3039 server_base.cc:1034] running on GCE node
W20260619 18:37:47.989585 3045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:47.990032 3039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:47.990717 3039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:47.991901 3039 hybrid_clock.cc:648] HybridClock initialized: now 1781894267991864 us; error 33 us; skew 500 ppm
Jun 19 18:37:47 dist-test-slave-rxv6 krb5kdc[2798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:47.994654 3039 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:37:47.995558 3039 webserver.cc:460] Webserver started at http://127.0.98.2:34239/ using document root <none> and password file <none>
I20260619 18:37:47.996101 3039 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:47.996167 3039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:47.996361 3039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:47.998437 3039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "0d93f7fe523543218e61555fd5811e0b"
format_stamp: "Formatted at 2026-06-19 18:37:47 on dist-test-slave-rxv6"
server_key: "f0b6ffb4a25c1f45ec5402cd8072789a"
server_key_iv: "eca4ddf10dddf6c4fe4427f87db085f9"
server_key_version: "encryptionkey@0"
I20260619 18:37:47.998997 3039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0d93f7fe523543218e61555fd5811e0b"
format_stamp: "Formatted at 2026-06-19 18:37:47 on dist-test-slave-rxv6"
server_key: "f0b6ffb4a25c1f45ec5402cd8072789a"
server_key_iv: "eca4ddf10dddf6c4fe4427f87db085f9"
server_key_version: "encryptionkey@0"
I20260619 18:37:48.002699 3039 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:37:48.005200 3054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:48.006494 3039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:37:48.006596 3039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "0d93f7fe523543218e61555fd5811e0b"
format_stamp: "Formatted at 2026-06-19 18:37:47 on dist-test-slave-rxv6"
server_key: "f0b6ffb4a25c1f45ec5402cd8072789a"
server_key_iv: "eca4ddf10dddf6c4fe4427f87db085f9"
server_key_version: "encryptionkey@0"
I20260619 18:37:48.006696 3039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:48.023901 3039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:48.026851 3039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:48.027042 3039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:48.027619 3039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:48.028429 3039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:48.028483 3039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:48.028537 3039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:48.028563 3039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:48.036949 3039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44169
I20260619 18:37:48.036991 3167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44169 every 8 connection(s)
I20260619 18:37:48.037832 3039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:37:48.046922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 3039
Jun 19 18:37:48 dist-test-slave-rxv6 krb5kdc[2798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:48.047019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:37:48.047271 392 external_mini_cluster.cc:1428] Setting key da9cd59e8876356fc67e28e7aa5852b0
I20260619 18:37:48.051205 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:48.039595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57153 (local address 127.0.98.62:43537)
0619 18:37:48.039717 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:48.039721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:48.040560 (+ 839us) server_negotiation.cc:407] Connection header received
0619 18:37:48.041981 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:48.041984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:48.042043 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:48.042122 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:48.043302 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:48.043831 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:48.044939 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:48.045161 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:48.045250 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:48.047595 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:48.047599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:48.047602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:48.047630 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:48.049100 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:48.049569 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:48.049573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:48.049575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:48.049659 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:48.050016 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:48.050019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:48.050021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:48.050241 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:48.050344 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:48.050976 (+ 632us) server_negotiation.cc:299] Negotiation successful
0619 18:37:48.051089 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:37:48.052151 3170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:48.039951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43537 (local address 127.0.98.2:57153)
0619 18:37:48.040396 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:48.040438 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:48.041743 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:48.042305 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:48.042312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:48.042675 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:48.043157 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:48.043170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:48.043944 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:48.043948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:48.044819 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:48.044828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:48.045295 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:48.045299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:48.045457 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:48.045988 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:48.046004 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:48.047404 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:48.049215 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:48.049219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:48.049232 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:48.049483 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:48.049739 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:48.049741 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:48.049743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:48.049865 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:48.050381 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:48.050388 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:48.050711 (+ 323us) client_negotiation.cc:769] Sending connection context
0619 18:37:48.050952 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 18:37:48.051303 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":112,"threads_started":1}
I20260619 18:37:48.053361 3168 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43537
I20260619 18:37:48.053634 3168 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:48.054245 3168 heartbeater.cc:511] Master 127.0.98.62:43537 requested a full tablet report, sending...
I20260619 18:37:48.055400 2843 ts_manager.cc:194] Registered new tserver with Master: 0d93f7fe523543218e61555fd5811e0b (127.0.98.2:44169)
I20260619 18:37:48.056241 2843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57153
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:37:48.073798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43537
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:48.180946 3173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:48.181183 3173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:48.181222 3173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:48.185197 3173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:48.185274 3173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:48.185351 3173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:37:48.190603 3173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:48.191566 3173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:48.192480 3173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:48.201557 3179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:48.201560 3178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:48.201629 3181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:48.201763 3173 server_base.cc:1034] running on GCE node
I20260619 18:37:48.202307 3173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:48.202966 3173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:48.204152 3173 hybrid_clock.cc:648] HybridClock initialized: now 1781894268204129 us; error 35 us; skew 500 ppm
Jun 19 18:37:48 dist-test-slave-rxv6 krb5kdc[2798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:48.206954 3173 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:37:48.207800 3173 webserver.cc:460] Webserver started at http://127.0.98.3:41449/ using document root <none> and password file <none>
I20260619 18:37:48.208329 3173 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:48.208393 3173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:48.208576 3173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:48.210669 3173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "98165d17cc3a4a448bd4f0b8ca06ac8e"
format_stamp: "Formatted at 2026-06-19 18:37:48 on dist-test-slave-rxv6"
server_key: "61b01b5e36bc4ee4c34d3113015ba082"
server_key_iv: "64a7e2c738f4c7b4b7ef3e1e714c0d1b"
server_key_version: "encryptionkey@0"
I20260619 18:37:48.211198 3173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "98165d17cc3a4a448bd4f0b8ca06ac8e"
format_stamp: "Formatted at 2026-06-19 18:37:48 on dist-test-slave-rxv6"
server_key: "61b01b5e36bc4ee4c34d3113015ba082"
server_key_iv: "64a7e2c738f4c7b4b7ef3e1e714c0d1b"
server_key_version: "encryptionkey@0"
I20260619 18:37:48.214787 3173 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:37:48.217082 3188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:48.218312 3173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:37:48.218413 3173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "98165d17cc3a4a448bd4f0b8ca06ac8e"
format_stamp: "Formatted at 2026-06-19 18:37:48 on dist-test-slave-rxv6"
server_key: "61b01b5e36bc4ee4c34d3113015ba082"
server_key_iv: "64a7e2c738f4c7b4b7ef3e1e714c0d1b"
server_key_version: "encryptionkey@0"
I20260619 18:37:48.218504 3173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:48.228510 3173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:48.231168 3173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:48.231333 3173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:48.231863 3173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:48.232640 3173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:48.232687 3173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:48.232738 3173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:48.232766 3173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:48.240778 3173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34521
I20260619 18:37:48.240813 3301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34521 every 8 connection(s)
I20260619 18:37:48.241616 3173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:37:48.249437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 3173
I20260619 18:37:48.249553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:37:48.249801 392 external_mini_cluster.cc:1428] Setting key 4b9a31741c9664cee9671b392b718aa8
Jun 19 18:37:48 dist-test-slave-rxv6 krb5kdc[2798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:48.254976 2882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:48.243475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48897 (local address 127.0.98.62:43537)
0619 18:37:48.243633 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:48.243637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:48.244499 (+ 862us) server_negotiation.cc:407] Connection header received
0619 18:37:48.245781 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:48.245785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:48.245834 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:48.245939 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:48.247145 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:48.247727 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:48.248850 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:48.249087 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:48.249184 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:48.251778 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:48.251781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:48.251784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:48.251812 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:48.253037 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:48.253540 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:48.253543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:48.253545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:48.253620 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:48.253946 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:48.253950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:48.253952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:48.254136 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:48.254220 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:48.254699 (+ 479us) server_negotiation.cc:299] Negotiation successful
0619 18:37:48.254817 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:37:48.255749 3304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:48.243826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43537 (local address 127.0.98.3:48897)
0619 18:37:48.244296 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:48.244336 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:48.245572 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:48.246115 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:48.246122 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:48.246461 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:48.246987 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:48.247004 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:48.247842 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:48.247844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:48.248725 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:48.248734 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:48.249266 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:48.249269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:48.249415 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:48.250022 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:48.250041 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:48.251618 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:48.253140 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:48.253146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:48.253163 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:48.253408 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:48.253705 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:48.253709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:48.253711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:48.253841 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:48.254228 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:48.254234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:48.254481 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 18:37:48.254646 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 18:37:48.254974 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":146,"threads_started":1}
I20260619 18:37:48.256860 3302 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43537
I20260619 18:37:48.257088 3302 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:48.257614 3302 heartbeater.cc:511] Master 127.0.98.62:43537 requested a full tablet report, sending...
I20260619 18:37:48.258737 2843 ts_manager.cc:194] Registered new tserver with Master: 98165d17cc3a4a448bd4f0b8ca06ac8e (127.0.98.3:34521)
I20260619 18:37:48.259310 2843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48897
I20260619 18:37:48.263649 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 18:37:48.275710 2882 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:37:48.274245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46840 (local address 127.0.98.62:43537)
0619 18:37:48.274372 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:48.274377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:48.274592 (+ 215us) server_negotiation.cc:407] Connection header received
0619 18:37:48.274717 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:48.274720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:48.274761 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:48.274827 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:48.275570 (+ 743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46840: BlockingRecv error: recv got EOF from 127.0.0.1:46840 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:37:48.277012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 2905
I20260619 18:37:48.281720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 3039
I20260619 18:37:48.286098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 3173
I20260619 18:37:48.290706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 2811
2026-06-19T18:37:48Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1990 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:37:48 dist-test-slave-rxv6 krb5kdc[3314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:37:48 dist-test-slave-rxv6 krb5kdc[3314](info): set up 2 sockets
Jun 19 18:37:48 dist-test-slave-rxv6 krb5kdc[3314](info): commencing operation
krb5kdc: starting...
W20260619 18:37:49.245751 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:37:49 dist-test-slave-rxv6 krb5kdc[3314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:37:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:37:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:37:49.360855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41983
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:49.468716 3327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:49.468973 3327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:49.469013 3327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:49.472970 3327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:37:49.473027 3327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:49.473045 3327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:49.473059 3327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:37:49.473078 3327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:37:49.478197 3327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:49.479172 3327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:49.480099 3327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:49.488131 3333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:49.488015 3335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:49.488004 3332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:49.488289 3327 server_base.cc:1034] running on GCE node
I20260619 18:37:49.488873 3327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:49.489952 3327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:49.491103 3327 hybrid_clock.cc:648] HybridClock initialized: now 1781894269491086 us; error 42 us; skew 500 ppm
Jun 19 18:37:49 dist-test-slave-rxv6 krb5kdc[3314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894269, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:49.493671 3327 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:37:49.494549 3327 webserver.cc:460] Webserver started at http://127.0.98.62:41661/ using document root <none> and password file <none>
I20260619 18:37:49.495065 3327 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:49.495133 3327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:49.495365 3327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:49.497323 3327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "7cebfbc7588242928e0d20a6de61f3fd"
format_stamp: "Formatted at 2026-06-19 18:37:49 on dist-test-slave-rxv6"
server_key: "9b2da5bf2dd8aff90dc01fddb85798ec"
server_key_iv: "b2a83a70faa036386d8a8264e334bf26"
server_key_version: "encryptionkey@0"
I20260619 18:37:49.497843 3327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "7cebfbc7588242928e0d20a6de61f3fd"
format_stamp: "Formatted at 2026-06-19 18:37:49 on dist-test-slave-rxv6"
server_key: "9b2da5bf2dd8aff90dc01fddb85798ec"
server_key_iv: "b2a83a70faa036386d8a8264e334bf26"
server_key_version: "encryptionkey@0"
I20260619 18:37:49.501330 3327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:37:49.503775 3342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:49.504969 3327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:37:49.505077 3327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "7cebfbc7588242928e0d20a6de61f3fd"
format_stamp: "Formatted at 2026-06-19 18:37:49 on dist-test-slave-rxv6"
server_key: "9b2da5bf2dd8aff90dc01fddb85798ec"
server_key_iv: "b2a83a70faa036386d8a8264e334bf26"
server_key_version: "encryptionkey@0"
I20260619 18:37:49.505167 3327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:49.545218 3327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:49.548548 3327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:49.548802 3327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:49.555612 3327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41983
I20260619 18:37:49.555859 3394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41983 every 8 connection(s)
I20260619 18:37:49.556568 3327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:37:49.557060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 3327
I20260619 18:37:49.557180 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:37:49.557449 392 external_mini_cluster.cc:1428] Setting key b1078f9507f285d327ea35f7927db2c6
I20260619 18:37:49.559626 3395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:49.565167 3395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd: Bootstrap starting.
Jun 19 18:37:49 dist-test-slave-rxv6 krb5kdc[3314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:49.567735 3395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:49.568432 3395 log.cc:824] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:49.570770 3395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd: No bootstrap required, opened a new log
I20260619 18:37:49.571625 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:49.558769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.0.98.62:41983)
0619 18:37:49.559230 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:49.559244 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:49.559281 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:37:49.559998 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:49.560017 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:49.560368 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:49.560691 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:49.561461 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:49.562492 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:49.563339 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:49.563806 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:49.563962 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:49.566606 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:49.566610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:49.566624 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:49.566653 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:49.568383 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:49.568845 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:49.568850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:49.568854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:49.568965 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:49.569270 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:49.569274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:49.569275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:49.569852 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:49.570093 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:49.570375 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 18:37:49.570683 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":142,"threads_started":1}
I20260619 18:37:49.574519 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:37:49.574936 3395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cebfbc7588242928e0d20a6de61f3fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41983 } }
I20260619 18:37:49.575165 3395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:49.575225 3395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cebfbc7588242928e0d20a6de61f3fd, State: Initialized, Role: FOLLOWER
I20260619 18:37:49.575640 3395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cebfbc7588242928e0d20a6de61f3fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41983 } }
I20260619 18:37:49.575762 3395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:37:49.575822 3395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:37:49.575914 3395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:49.577049 3395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cebfbc7588242928e0d20a6de61f3fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41983 } }
I20260619 18:37:49.577419 3395 leader_election.cc:302] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cebfbc7588242928e0d20a6de61f3fd; no voters:
I20260619 18:37:49.577744 3395 leader_election.cc:288] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:37:49.577847 3401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:49.578076 3401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [term 1 LEADER]: Becoming Leader. State: Replica: 7cebfbc7588242928e0d20a6de61f3fd, State: Running, Role: LEADER
I20260619 18:37:49.578485 3401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cebfbc7588242928e0d20a6de61f3fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41983 } }
I20260619 18:37:49.578974 3395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:37:49.579738 3401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cebfbc7588242928e0d20a6de61f3fd. Latest consensus state: current_term: 1 leader_uuid: "7cebfbc7588242928e0d20a6de61f3fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cebfbc7588242928e0d20a6de61f3fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41983 } } }
I20260619 18:37:49.579849 3401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:49.580435 3402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cebfbc7588242928e0d20a6de61f3fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cebfbc7588242928e0d20a6de61f3fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41983 } } }
I20260619 18:37:49.580538 3402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:49.580595 3407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:37:49.583887 3407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:37:49.589808 3407 catalog_manager.cc:1347] Generated new cluster ID: fca2bfd851ea440b9faaea348e8fdc71
I20260619 18:37:49.589874 3407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:37:49.593607 3398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:49.587621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.0.98.62:41983)
0619 18:37:49.587739 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:49.587743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:49.588124 (+ 381us) server_negotiation.cc:407] Connection header received
0619 18:37:49.588536 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:49.588540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:49.588591 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:49.588683 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:49.589294 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:49.590120 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:49.591289 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:49.591657 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:49.591765 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:49.592632 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:49.592636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:49.592640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:37:49.593180 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:49.593215 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:37:49.593235 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:49.593326 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:49.593402 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 18:37:49.593465 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:37:49.598747 3407 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:37:49.599895 3407 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:37:49.609401 3407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cebfbc7588242928e0d20a6de61f3fd: Generated new TSK 0
I20260619 18:37:49.610206 3407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:37:49.619201 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 3327
2026-06-19T18:37:49Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1327 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:37:49 dist-test-slave-rxv6 krb5kdc[3428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:37:49 dist-test-slave-rxv6 krb5kdc[3428](info): set up 2 sockets
Jun 19 18:37:49 dist-test-slave-rxv6 krb5kdc[3428](info): commencing operation
krb5kdc: starting...
W20260619 18:37:50.573805 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:37:50 dist-test-slave-rxv6 krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:37:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:37:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:37:50.689646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38645
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:50.796788 3441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:50.797072 3441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:50.797113 3441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:50.801127 3441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:37:50.801194 3441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:50.801213 3441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:50.801229 3441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:37:50.801241 3441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:37:50.806469 3441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:50.807443 3441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:50.808387 3441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:50.816670 3449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:50.816710 3447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:50.816679 3446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:50.816792 3441 server_base.cc:1034] running on GCE node
I20260619 18:37:50.817447 3441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:50.818655 3441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:50.819813 3441 hybrid_clock.cc:648] HybridClock initialized: now 1781894270819810 us; error 59 us; skew 500 ppm
Jun 19 18:37:50 dist-test-slave-rxv6 krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:50.822440 3441 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:37:50.823381 3441 webserver.cc:460] Webserver started at http://127.0.98.62:40523/ using document root <none> and password file <none>
I20260619 18:37:50.823906 3441 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:50.823971 3441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:50.824165 3441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:50.826224 3441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "ff3db569b58e4c019ee30fd023c0c2ba"
format_stamp: "Formatted at 2026-06-19 18:37:50 on dist-test-slave-rxv6"
server_key: "343af210afbb94310ba955e0cadd2f91"
server_key_iv: "1e8961442887eb7c77c1644befcf504f"
server_key_version: "encryptionkey@0"
I20260619 18:37:50.826766 3441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "ff3db569b58e4c019ee30fd023c0c2ba"
format_stamp: "Formatted at 2026-06-19 18:37:50 on dist-test-slave-rxv6"
server_key: "343af210afbb94310ba955e0cadd2f91"
server_key_iv: "1e8961442887eb7c77c1644befcf504f"
server_key_version: "encryptionkey@0"
I20260619 18:37:50.830446 3441 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:37:50.832821 3456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:50.834091 3441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:37:50.834201 3441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "ff3db569b58e4c019ee30fd023c0c2ba"
format_stamp: "Formatted at 2026-06-19 18:37:50 on dist-test-slave-rxv6"
server_key: "343af210afbb94310ba955e0cadd2f91"
server_key_iv: "1e8961442887eb7c77c1644befcf504f"
server_key_version: "encryptionkey@0"
I20260619 18:37:50.834347 3441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:50.847018 3441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:50.849826 3441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:50.850037 3441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:50.856419 3441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38645
I20260619 18:37:50.856665 3508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38645 every 8 connection(s)
I20260619 18:37:50.857331 3441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:37:50.859982 3509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:50.865178 3509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba: Bootstrap starting.
I20260619 18:37:50.865459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 3441
I20260619 18:37:50.865562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:37:50.865793 392 external_mini_cluster.cc:1428] Setting key 1e10d83a8591be1b21837fcae0f705bb
I20260619 18:37:50.867578 3509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:50.868247 3509 log.cc:824] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:50.870245 3509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba: No bootstrap required, opened a new log
I20260619 18:37:50.874329 3509 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3db569b58e4c019ee30fd023c0c2ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } }
I20260619 18:37:50.874555 3509 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:50.874606 3509 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff3db569b58e4c019ee30fd023c0c2ba, State: Initialized, Role: FOLLOWER
Jun 19 18:37:50 dist-test-slave-rxv6 krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:50.875099 3509 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3db569b58e4c019ee30fd023c0c2ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } }
I20260619 18:37:50.875258 3509 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:37:50.875325 3509 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:37:50.875418 3509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:50.876238 3509 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3db569b58e4c019ee30fd023c0c2ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } }
I20260619 18:37:50.876576 3509 leader_election.cc:302] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff3db569b58e4c019ee30fd023c0c2ba; no voters:
I20260619 18:37:50.876924 3509 leader_election.cc:288] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:37:50.876986 3514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:50.877177 3514 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [term 1 LEADER]: Becoming Leader. State: Replica: ff3db569b58e4c019ee30fd023c0c2ba, State: Running, Role: LEADER
I20260619 18:37:50.877518 3514 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3db569b58e4c019ee30fd023c0c2ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } }
I20260619 18:37:50.878224 3509 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:37:50.879187 3515 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff3db569b58e4c019ee30fd023c0c2ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3db569b58e4c019ee30fd023c0c2ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } } }
I20260619 18:37:50.879184 3516 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff3db569b58e4c019ee30fd023c0c2ba. Latest consensus state: current_term: 1 leader_uuid: "ff3db569b58e4c019ee30fd023c0c2ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff3db569b58e4c019ee30fd023c0c2ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38645 } } }
I20260619 18:37:50.879313 3515 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:50.879313 3516 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:50.879712 3523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:37:50.881057 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:50.867030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.0.98.62:38645)
0619 18:37:50.867412 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:50.867427 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:50.867461 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:37:50.868103 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:50.868122 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:50.868463 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:50.868808 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:50.869594 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:50.870728 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:50.871881 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:50.872353 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:50.872474 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:50.875289 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:50.875294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:50.875308 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:50.875371 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:50.877294 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:50.877759 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:50.877765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:50.877771 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:50.877930 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:50.878213 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:50.878217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:50.878219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:50.878777 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:50.878952 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:50.879789 (+ 837us) server_negotiation.cc:299] Negotiation successful
0619 18:37:50.880102 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
I20260619 18:37:50.882797 3523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:37:50.887671 3523 catalog_manager.cc:1347] Generated new cluster ID: 3ac7098ea9fe4c21a3b201f55440e01b
I20260619 18:37:50.887730 3523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:37:50.905990 3523 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:37:50.907009 3523 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:37:50.912202 3523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff3db569b58e4c019ee30fd023c0c2ba: Generated new TSK 0
I20260619 18:37:50.912915 3523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:37:50.944783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38645
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:51.072118 3535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:51.072467 3535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:51.072516 3535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:51.076572 3535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:51.076632 3535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:51.076690 3535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:37:51.082126 3535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:51.083159 3535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:51.084121 3535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:51.093228 3541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:51.093228 3540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:51.093330 3543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:51.093672 3535 server_base.cc:1034] running on GCE node
I20260619 18:37:51.094089 3535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:51.094730 3535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:51.095877 3535 hybrid_clock.cc:648] HybridClock initialized: now 1781894271095859 us; error 30 us; skew 500 ppm
Jun 19 18:37:51 dist-test-slave-rxv6 krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:51.098647 3535 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:37:51.099500 3535 webserver.cc:460] Webserver started at http://127.0.98.1:38735/ using document root <none> and password file <none>
I20260619 18:37:51.100036 3535 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:51.100103 3535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:51.100272 3535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:51.102479 3535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "457585e458f04b0fbb1b25c87b8d71ad"
format_stamp: "Formatted at 2026-06-19 18:37:51 on dist-test-slave-rxv6"
server_key: "bbf2a240af19090d49c96c43426ada5a"
server_key_iv: "6ef86ce2520ec4bc713ccbe7bfbc344b"
server_key_version: "encryptionkey@0"
I20260619 18:37:51.102980 3535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "457585e458f04b0fbb1b25c87b8d71ad"
format_stamp: "Formatted at 2026-06-19 18:37:51 on dist-test-slave-rxv6"
server_key: "bbf2a240af19090d49c96c43426ada5a"
server_key_iv: "6ef86ce2520ec4bc713ccbe7bfbc344b"
server_key_version: "encryptionkey@0"
I20260619 18:37:51.106467 3535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:37:51.108796 3550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:51.110116 3535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:37:51.110206 3535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "457585e458f04b0fbb1b25c87b8d71ad"
format_stamp: "Formatted at 2026-06-19 18:37:51 on dist-test-slave-rxv6"
server_key: "bbf2a240af19090d49c96c43426ada5a"
server_key_iv: "6ef86ce2520ec4bc713ccbe7bfbc344b"
server_key_version: "encryptionkey@0"
I20260619 18:37:51.110287 3535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:51.121083 3535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:51.124086 3535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:51.124228 3535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:51.124750 3535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:51.125536 3535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:51.125581 3535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:51.125622 3535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:51.125643 3535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:51.133533 3535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36417
I20260619 18:37:51.133594 3663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36417 every 8 connection(s)
I20260619 18:37:51.134423 3535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:37:51.141405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 3535
I20260619 18:37:51.141520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:37:51.141786 392 external_mini_cluster.cc:1428] Setting key 91d8886a8533232763e346696840f070
Jun 19 18:37:51 dist-test-slave-rxv6 krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:51.147766 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:51.136068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32981 (local address 127.0.98.62:38645)
0619 18:37:51.136200 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:51.136205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:51.137010 (+ 805us) server_negotiation.cc:407] Connection header received
0619 18:37:51.138432 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:51.138435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:51.138481 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:51.138552 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:51.139712 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:51.140237 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:51.141438 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:51.141721 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:51.141797 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:51.144596 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:51.144599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:51.144601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:51.144623 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:51.145946 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:51.146389 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:51.146393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:51.146395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:51.146467 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:51.146755 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:51.146758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:51.146760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:51.146935 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:51.147016 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:51.147485 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 18:37:51.147610 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:37:51.148530 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:51.136461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38645 (local address 127.0.98.1:32981)
0619 18:37:51.136868 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:51.136911 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:51.138223 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:51.138762 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:51.138770 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:51.139123 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:51.139591 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:51.139604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:51.140351 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:51.140354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:51.141331 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:51.141342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:51.141860 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:51.141865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:51.142091 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:51.142939 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:51.142955 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:51.144448 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:51.146041 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:51.146047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:51.146062 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:51.146301 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:51.146543 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:51.146546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:51.146548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:51.146676 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:51.147026 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:51.147032 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:51.147265 (+ 233us) client_negotiation.cc:769] Sending connection context
0619 18:37:51.147430 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 18:37:51.147734 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260619 18:37:51.149755 3664 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38645
I20260619 18:37:51.150027 3664 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:51.150467 3664 heartbeater.cc:511] Master 127.0.98.62:38645 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 18:37:51.151897 3473 ts_manager.cc:194] Registered new tserver with Master: 457585e458f04b0fbb1b25c87b8d71ad (127.0.98.1:36417)
I20260619 18:37:51.152981 3473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:37:51.168030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38645
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:51.274915 3669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:51.275202 3669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:51.275244 3669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:51.279263 3669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:51.279325 3669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:51.279392 3669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:37:51.284665 3669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:51.285645 3669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:51.286648 3669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:51.296157 3674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:51.296324 3669 server_base.cc:1034] running on GCE node
W20260619 18:37:51.296146 3675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:51.296144 3677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:51.296793 3669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:51.297410 3669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:51.298563 3669 hybrid_clock.cc:648] HybridClock initialized: now 1781894271298534 us; error 41 us; skew 500 ppm
Jun 19 18:37:51 dist-test-slave-rxv6 krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:51.301143 3669 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:37:51.301970 3669 webserver.cc:460] Webserver started at http://127.0.98.2:34811/ using document root <none> and password file <none>
I20260619 18:37:51.302484 3669 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:51.302541 3669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:51.302697 3669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:51.304697 3669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "ac26202fae2747ddbf1fe53c43539fd9"
format_stamp: "Formatted at 2026-06-19 18:37:51 on dist-test-slave-rxv6"
server_key: "d8e14bc62421e4ea92ea2f4893795e6d"
server_key_iv: "088a607e1a7226dc9d0bea5283df125c"
server_key_version: "encryptionkey@0"
I20260619 18:37:51.305193 3669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ac26202fae2747ddbf1fe53c43539fd9"
format_stamp: "Formatted at 2026-06-19 18:37:51 on dist-test-slave-rxv6"
server_key: "d8e14bc62421e4ea92ea2f4893795e6d"
server_key_iv: "088a607e1a7226dc9d0bea5283df125c"
server_key_version: "encryptionkey@0"
I20260619 18:37:51.308758 3669 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:37:51.311091 3684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:51.312227 3669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:37:51.312318 3669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "ac26202fae2747ddbf1fe53c43539fd9"
format_stamp: "Formatted at 2026-06-19 18:37:51 on dist-test-slave-rxv6"
server_key: "d8e14bc62421e4ea92ea2f4893795e6d"
server_key_iv: "088a607e1a7226dc9d0bea5283df125c"
server_key_version: "encryptionkey@0"
I20260619 18:37:51.312418 3669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:51.335677 3669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:51.338765 3669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:51.338934 3669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:51.339512 3669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:51.340276 3669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:51.340319 3669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:51.340355 3669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:51.340375 3669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:51.348349 3669 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45159
I20260619 18:37:51.348376 3797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45159 every 8 connection(s)
I20260619 18:37:51.349179 3669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:37:51.353858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 3669
I20260619 18:37:51.354020 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:37:51.354269 392 external_mini_cluster.cc:1428] Setting key f2cb61ec0e0bcec0b8c00562b9537447
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 19 18:37:51 dist-test-slave-rxv6 krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:51.371546 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:51.350862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36017 (local address 127.0.98.62:38645)
0619 18:37:51.351037 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:51.351042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:51.351843 (+ 801us) server_negotiation.cc:407] Connection header received
0619 18:37:51.353194 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:51.353197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:51.353241 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:51.353314 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:51.354530 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:51.355124 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:51.356307 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:51.356611 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:51.356716 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:51.367717 (+ 11001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:51.367721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:51.367724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:51.367757 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:51.369283 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:51.369813 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:51.369817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:51.369819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:51.369946 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:51.370345 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:51.370348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:51.370350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:51.370595 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:51.370694 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:51.371239 (+ 545us) server_negotiation.cc:299] Negotiation successful
0619 18:37:51.371363 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 18:37:51.372316 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:51.351284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38645 (local address 127.0.98.2:36017)
0619 18:37:51.351704 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:51.351744 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:51.352970 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:51.353471 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:51.353478 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:51.353814 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:51.354400 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:51.354415 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:51.355235 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:51.355238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:51.356191 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:51.356200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:51.356765 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:51.356769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:51.356949 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:51.365689 (+ 8740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:51.365714 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:51.367493 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:51.369429 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:51.369435 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:51.369456 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:51.369716 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:51.370094 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:51.370097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:51.370099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:51.370250 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:51.370707 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:51.370713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:51.371002 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 18:37:51.371168 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 18:37:51.371441 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260619 18:37:51.373375 3798 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38645
I20260619 18:37:51.373615 3798 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:51.374169 3798 heartbeater.cc:511] Master 127.0.98.62:38645 requested a full tablet report, sending...
I20260619 18:37:51.375193 3473 ts_manager.cc:194] Registered new tserver with Master: ac26202fae2747ddbf1fe53c43539fd9 (127.0.98.2:45159)
I20260619 18:37:51.375758 3473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:37:51.380282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38645
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:51.487112 3803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:51.487375 3803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:51.487417 3803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:51.491333 3803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:51.491385 3803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:51.491442 3803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:37:51.496793 3803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:51.497759 3803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:51.498764 3803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:51.507862 3811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:51.507829 3809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:51.507980 3803 server_base.cc:1034] running on GCE node
W20260619 18:37:51.508149 3808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:51.508517 3803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:51.509099 3803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:51.510264 3803 hybrid_clock.cc:648] HybridClock initialized: now 1781894271510246 us; error 32 us; skew 500 ppm
Jun 19 18:37:51 dist-test-slave-rxv6 krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:51.513008 3803 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:37:51.513870 3803 webserver.cc:460] Webserver started at http://127.0.98.3:37911/ using document root <none> and password file <none>
I20260619 18:37:51.514451 3803 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:51.514519 3803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:51.514719 3803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:51.516901 3803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "91be6b567dfa41ff83f50cb05d16dceb"
format_stamp: "Formatted at 2026-06-19 18:37:51 on dist-test-slave-rxv6"
server_key: "ec7a8382d5af844c529c63c5c3b4087b"
server_key_iv: "51b921f7bc53b3630f930a1d8ec09e95"
server_key_version: "encryptionkey@0"
I20260619 18:37:51.517431 3803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "91be6b567dfa41ff83f50cb05d16dceb"
format_stamp: "Formatted at 2026-06-19 18:37:51 on dist-test-slave-rxv6"
server_key: "ec7a8382d5af844c529c63c5c3b4087b"
server_key_iv: "51b921f7bc53b3630f930a1d8ec09e95"
server_key_version: "encryptionkey@0"
I20260619 18:37:51.520958 3803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 18:37:51.523317 3818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:51.524587 3803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:37:51.524775 3803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "91be6b567dfa41ff83f50cb05d16dceb"
format_stamp: "Formatted at 2026-06-19 18:37:51 on dist-test-slave-rxv6"
server_key: "ec7a8382d5af844c529c63c5c3b4087b"
server_key_iv: "51b921f7bc53b3630f930a1d8ec09e95"
server_key_version: "encryptionkey@0"
I20260619 18:37:51.524883 3803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:51.543040 3803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:51.545750 3803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:51.545940 3803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:51.546505 3803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:51.547283 3803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:51.547329 3803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:51.547379 3803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:51.547408 3803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:51.555377 3803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35819
I20260619 18:37:51.555398 3931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35819 every 8 connection(s)
I20260619 18:37:51.556202 3803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
Jun 19 18:37:51 dist-test-slave-rxv6 krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:51.566156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 3803
I20260619 18:37:51.566267 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:37:51.566597 392 external_mini_cluster.cc:1428] Setting key c650a9a8ff85ae6678b649efe99e2251
I20260619 18:37:51.569195 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:51.557887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40551 (local address 127.0.98.62:38645)
0619 18:37:51.558040 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:51.558044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:51.558771 (+ 727us) server_negotiation.cc:407] Connection header received
0619 18:37:51.560113 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:51.560116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:51.560169 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:51.560241 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:51.561351 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:51.561869 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:51.562983 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:51.563199 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:51.563284 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:51.565423 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:51.565426 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:51.565429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:51.565455 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:51.566945 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:51.567498 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:51.567501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:51.567503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:51.567589 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:51.567982 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:51.567986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:51.567987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:51.568188 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:51.568265 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:51.568973 (+ 708us) server_negotiation.cc:299] Negotiation successful
0619 18:37:51.569078 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 18:37:51.569895 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:51.558239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38645 (local address 127.0.98.3:40551)
0619 18:37:51.558636 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:51.558676 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:51.559913 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:51.560414 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:51.560428 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:51.560767 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:51.561215 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:51.561228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:51.562006 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:51.562010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:51.562865 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:51.562874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:51.563325 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:51.563329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:51.563488 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:51.563949 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:51.563964 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:51.565269 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:51.567073 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:51.567079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:51.567094 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:51.567414 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:51.567679 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:51.567682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:51.567683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:51.567838 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:51.568307 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:51.568314 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:51.568627 (+ 313us) client_negotiation.cc:769] Sending connection context
0619 18:37:51.568838 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 18:37:51.569102 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":98,"threads_started":1}
I20260619 18:37:51.570864 3932 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38645
I20260619 18:37:51.571095 3932 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:51.571517 3932 heartbeater.cc:511] Master 127.0.98.62:38645 requested a full tablet report, sending...
I20260619 18:37:51.572387 3473 ts_manager.cc:194] Registered new tserver with Master: 91be6b567dfa41ff83f50cb05d16dceb (127.0.98.3:35819)
I20260619 18:37:51.572918 3473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40551
I20260619 18:37:51.580402 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:37:51 dist-test-slave-rxv6 krb5kdc[3428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894271, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 18:37:51 dist-test-slave-rxv6 krb5kdc[3428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894271, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:51.604794 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:51.596271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.0.98.62:38645)
0619 18:37:51.596455 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:51.596460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:51.596533 (+ 73us) server_negotiation.cc:407] Connection header received
0619 18:37:51.596642 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:51.596645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:51.596688 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:51.596756 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:51.597402 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:51.597963 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:51.598717 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:51.598924 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:51.599005 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:51.601253 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:51.601256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:51.601258 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:51.601281 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:51.602660 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:51.603611 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:51.603615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:51.603616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:51.603695 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:51.603949 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:51.603952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:51.603954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:51.604181 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:51.604303 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:51.604524 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 18:37:51.604638 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 18:37:51.605154 3473 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50544
I20260619 18:37:51.605993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 3535
I20260619 18:37:51.610729 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 3669
I20260619 18:37:51.615577 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 3803
I20260619 18:37:51.619964 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 3441
2026-06-19T18:37:51Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2002 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:37:51 dist-test-slave-rxv6 krb5kdc[3944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:37:51 dist-test-slave-rxv6 krb5kdc[3944](info): set up 2 sockets
Jun 19 18:37:51 dist-test-slave-rxv6 krb5kdc[3944](info): commencing operation
krb5kdc: starting...
W20260619 18:37:52.577710 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:37:52 dist-test-slave-rxv6 krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:37:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:37:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:37:52.692416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44769
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:52.799015 3957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:52.799314 3957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:52.799371 3957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:52.803349 3957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:37:52.803401 3957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:52.803428 3957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:52.803448 3957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:37:52.803469 3957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:37:52.808627 3957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:52.809613 3957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:52.810603 3957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:52.818370 3965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:52.818377 3963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:52.818435 3957 server_base.cc:1034] running on GCE node
W20260619 18:37:52.818411 3962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:52.819041 3957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:52.820062 3957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:52.821229 3957 hybrid_clock.cc:648] HybridClock initialized: now 1781894272821215 us; error 29 us; skew 500 ppm
Jun 19 18:37:52 dist-test-slave-rxv6 krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:52.823884 3957 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:37:52.824697 3957 webserver.cc:460] Webserver started at http://127.0.98.62:46785/ using document root <none> and password file <none>
I20260619 18:37:52.825279 3957 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:52.825348 3957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:52.825527 3957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:52.827574 3957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "f5d1c6ce8d3841c3a946b57c421fef36"
format_stamp: "Formatted at 2026-06-19 18:37:52 on dist-test-slave-rxv6"
server_key: "68348ac54017a4ddcc70bc167a52b760"
server_key_iv: "67c48e10e38c4582b174090f553620a8"
server_key_version: "encryptionkey@0"
I20260619 18:37:52.828118 3957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "f5d1c6ce8d3841c3a946b57c421fef36"
format_stamp: "Formatted at 2026-06-19 18:37:52 on dist-test-slave-rxv6"
server_key: "68348ac54017a4ddcc70bc167a52b760"
server_key_iv: "67c48e10e38c4582b174090f553620a8"
server_key_version: "encryptionkey@0"
I20260619 18:37:52.831717 3957 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 18:37:52.833995 3972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:52.835151 3957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:37:52.835268 3957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "f5d1c6ce8d3841c3a946b57c421fef36"
format_stamp: "Formatted at 2026-06-19 18:37:52 on dist-test-slave-rxv6"
server_key: "68348ac54017a4ddcc70bc167a52b760"
server_key_iv: "67c48e10e38c4582b174090f553620a8"
server_key_version: "encryptionkey@0"
I20260619 18:37:52.835355 3957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:52.855592 3957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:52.858548 3957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:52.858724 3957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:52.865366 3957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44769
I20260619 18:37:52.865392 4024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44769 every 8 connection(s)
I20260619 18:37:52.866325 3957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:37:52.868134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 3957
I20260619 18:37:52.868251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:37:52.868496 392 external_mini_cluster.cc:1428] Setting key 421ea0ef6a3d8ef7e65a963c50789d4a
I20260619 18:37:52.869096 4025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:52.875674 4025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36: Bootstrap starting.
Jun 19 18:37:52 dist-test-slave-rxv6 krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:52.878207 4025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:52.879041 4025 log.cc:824] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:52.881151 4025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36: No bootstrap required, opened a new log
I20260619 18:37:52.882452 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:52.869772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42912 (local address 127.0.98.62:44769)
0619 18:37:52.870197 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:52.870215 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:52.870256 (+ 41us) server_negotiation.cc:407] Connection header received
0619 18:37:52.871075 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:52.871108 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:52.871516 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:52.871879 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:52.872533 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:52.874008 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:52.874825 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:52.875473 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:52.875623 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:52.877771 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:52.877777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:52.877793 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:52.877823 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:52.879612 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:52.880004 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:52.880009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:52.880015 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:52.880193 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:52.880406 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:52.880409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:52.880411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:52.880878 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:52.881047 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:52.881235 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 18:37:52.881516 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":130,"threads_started":1}
I20260619 18:37:52.884763 4025 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1c6ce8d3841c3a946b57c421fef36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44769 } }
I20260619 18:37:52.885015 4025 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:52.885058 4025 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d1c6ce8d3841c3a946b57c421fef36, State: Initialized, Role: FOLLOWER
I20260619 18:37:52.885397 4025 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1c6ce8d3841c3a946b57c421fef36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44769 } }
I20260619 18:37:52.885499 4025 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:37:52.885540 4025 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:37:52.885614 4025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:52.886469 4025 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1c6ce8d3841c3a946b57c421fef36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44769 } }
I20260619 18:37:52.886759 4025 leader_election.cc:302] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5d1c6ce8d3841c3a946b57c421fef36; no voters:
I20260619 18:37:52.887005 4025 leader_election.cc:288] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:37:52.887099 4030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:52.887310 4030 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [term 1 LEADER]: Becoming Leader. State: Replica: f5d1c6ce8d3841c3a946b57c421fef36, State: Running, Role: LEADER
I20260619 18:37:52.887612 4030 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1c6ce8d3841c3a946b57c421fef36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44769 } }
I20260619 18:37:52.888060 4025 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:37:52.889165 4032 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5d1c6ce8d3841c3a946b57c421fef36. Latest consensus state: current_term: 1 leader_uuid: "f5d1c6ce8d3841c3a946b57c421fef36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1c6ce8d3841c3a946b57c421fef36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44769 } } }
I20260619 18:37:52.889200 4031 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5d1c6ce8d3841c3a946b57c421fef36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d1c6ce8d3841c3a946b57c421fef36" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44769 } } }
I20260619 18:37:52.889293 4032 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:52.889317 4031 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36 [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:52.889605 4039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:37:52.892957 4039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:37:52.898303 4039 catalog_manager.cc:1347] Generated new cluster ID: 2b74547d30224f35bca75b4005464f3c
I20260619 18:37:52.898376 4039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:37:52.926910 4039 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:37:52.927912 4039 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:37:52.935868 4039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5d1c6ce8d3841c3a946b57c421fef36: Generated new TSK 0
I20260619 18:37:52.936726 4039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:37:52.970937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44769
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:53.081329 4051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:53.081617 4051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:53.081664 4051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:53.085813 4051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:53.085882 4051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:53.085979 4051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:37:53.091396 4051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:53.092409 4051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:53.093382 4051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:53.102615 4057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:53.102797 4051 server_base.cc:1034] running on GCE node
W20260619 18:37:53.102610 4056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:53.102615 4059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:53.103685 4051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:53.104316 4051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:53.105489 4051 hybrid_clock.cc:648] HybridClock initialized: now 1781894273105462 us; error 39 us; skew 500 ppm
Jun 19 18:37:53 dist-test-slave-rxv6 krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:53.108206 4051 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:37:53.109081 4051 webserver.cc:460] Webserver started at http://127.0.98.1:43989/ using document root <none> and password file <none>
I20260619 18:37:53.109616 4051 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:53.109675 4051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:53.109865 4051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:53.111876 4051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "0839596f9ce54c61a9f33c811ca82290"
format_stamp: "Formatted at 2026-06-19 18:37:53 on dist-test-slave-rxv6"
server_key: "6eecf481d354cac4c1c966facec8dc41"
server_key_iv: "0a234ea9dbf7164dbd3ac42bbaba337e"
server_key_version: "encryptionkey@0"
I20260619 18:37:53.112388 4051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0839596f9ce54c61a9f33c811ca82290"
format_stamp: "Formatted at 2026-06-19 18:37:53 on dist-test-slave-rxv6"
server_key: "6eecf481d354cac4c1c966facec8dc41"
server_key_iv: "0a234ea9dbf7164dbd3ac42bbaba337e"
server_key_version: "encryptionkey@0"
I20260619 18:37:53.115938 4051 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:37:53.118276 4066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:53.119424 4051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:37:53.119529 4051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "0839596f9ce54c61a9f33c811ca82290"
format_stamp: "Formatted at 2026-06-19 18:37:53 on dist-test-slave-rxv6"
server_key: "6eecf481d354cac4c1c966facec8dc41"
server_key_iv: "0a234ea9dbf7164dbd3ac42bbaba337e"
server_key_version: "encryptionkey@0"
I20260619 18:37:53.119624 4051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:53.130115 4051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:53.132893 4051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:53.133057 4051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:53.133606 4051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:53.134441 4051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:53.134486 4051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:53.134528 4051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:53.134557 4051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:53.142877 4051 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36681
I20260619 18:37:53.142948 4179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36681 every 8 connection(s)
I20260619 18:37:53.143756 4051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:37:53.147351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 4051
I20260619 18:37:53.147472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:37:53.147751 392 external_mini_cluster.cc:1428] Setting key 44c6deabf97ee0eeebe34cd0e4e2f66b
Jun 19 18:37:53 dist-test-slave-rxv6 krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 18:37:53.157537 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:53.145502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42441 (local address 127.0.98.62:44769)
0619 18:37:53.145643 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:53.145648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:53.146482 (+ 834us) server_negotiation.cc:407] Connection header received
0619 18:37:53.147909 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:53.147913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:53.147969 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:53.148047 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:53.149366 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:53.149885 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:53.151237 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:53.151469 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:53.151558 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:53.153899 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:53.153903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:53.153934 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:53.153960 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:53.155467 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:53.156011 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:53.156014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:53.156016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:53.156090 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:53.156427 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:53.156430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:53.156432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:53.156642 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:53.156733 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:53.157237 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 18:37:53.157368 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 18:37:53.158661 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:53.145876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44769 (local address 127.0.98.1:42441)
0619 18:37:53.146338 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:53.146378 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:53.147704 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:53.148237 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:53.148245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:53.148665 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:53.149234 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:53.149249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:53.150024 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:53.150028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:53.151119 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:53.151128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:53.151603 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:53.151607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:53.151778 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:53.152313 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:53.152335 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:53.153738 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:53.155602 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:53.155609 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:53.155626 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:53.155912 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:53.156195 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:53.156198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:53.156200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:53.156339 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:53.156738 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:53.156745 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:53.157001 (+ 256us) client_negotiation.cc:769] Sending connection context
0619 18:37:53.157181 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 18:37:53.157602 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
I20260619 18:37:53.160648 4180 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44769
I20260619 18:37:53.160902 4180 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:53.161401 4180 heartbeater.cc:511] Master 127.0.98.62:44769 requested a full tablet report, sending...
I20260619 18:37:53.162919 3989 ts_manager.cc:194] Registered new tserver with Master: 0839596f9ce54c61a9f33c811ca82290 (127.0.98.1:36681)
I20260619 18:37:53.164083 3989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:37:53.172911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44769
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:53.281950 4185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:53.282225 4185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:53.282267 4185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:53.286293 4185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:53.286355 4185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:53.286417 4185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:37:53.291690 4185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:53.292732 4185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:53.293746 4185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:53.303380 4190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:53.303804 4193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:53.303874 4191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:53.303951 4185 server_base.cc:1034] running on GCE node
I20260619 18:37:53.304350 4185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:53.304968 4185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:53.306123 4185 hybrid_clock.cc:648] HybridClock initialized: now 1781894273306096 us; error 36 us; skew 500 ppm
Jun 19 18:37:53 dist-test-slave-rxv6 krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:53.308822 4185 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:37:53.309743 4185 webserver.cc:460] Webserver started at http://127.0.98.2:37653/ using document root <none> and password file <none>
I20260619 18:37:53.310568 4185 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:53.310671 4185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:53.310887 4185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:53.312970 4185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "928c605f47aa46aca9f92c5a92654047"
format_stamp: "Formatted at 2026-06-19 18:37:53 on dist-test-slave-rxv6"
server_key: "3fd5dc368e6e93719c6add7f02c0f14f"
server_key_iv: "253205f65f34d232e87c0afd0ccf199a"
server_key_version: "encryptionkey@0"
I20260619 18:37:53.313506 4185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "928c605f47aa46aca9f92c5a92654047"
format_stamp: "Formatted at 2026-06-19 18:37:53 on dist-test-slave-rxv6"
server_key: "3fd5dc368e6e93719c6add7f02c0f14f"
server_key_iv: "253205f65f34d232e87c0afd0ccf199a"
server_key_version: "encryptionkey@0"
I20260619 18:37:53.317255 4185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:37:53.319815 4200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:53.321030 4185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 18:37:53.321139 4185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "928c605f47aa46aca9f92c5a92654047"
format_stamp: "Formatted at 2026-06-19 18:37:53 on dist-test-slave-rxv6"
server_key: "3fd5dc368e6e93719c6add7f02c0f14f"
server_key_iv: "253205f65f34d232e87c0afd0ccf199a"
server_key_version: "encryptionkey@0"
I20260619 18:37:53.321237 4185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:53.341267 4185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:53.344506 4185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:53.344686 4185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:53.345268 4185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:53.346199 4185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:53.346248 4185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:53.346299 4185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:53.346323 4185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:53.354671 4185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35575
I20260619 18:37:53.354758 4313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35575 every 8 connection(s)
I20260619 18:37:53.355548 4185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:37:53.358958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 4185
I20260619 18:37:53.359053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:37:53.359301 392 external_mini_cluster.cc:1428] Setting key 15fff61ca444b95bb640f75528eadb65
Jun 19 18:37:53 dist-test-slave-rxv6 krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 18:37:53.370180 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:53.357326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58851 (local address 127.0.98.62:44769)
0619 18:37:53.357470 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:53.357473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:53.358279 (+ 806us) server_negotiation.cc:407] Connection header received
0619 18:37:53.359738 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:53.359741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:53.359794 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:53.359886 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:53.361402 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:53.362138 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:53.363713 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:53.363929 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:53.364006 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:53.366645 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:53.366650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:53.366654 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:53.366685 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:53.368258 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:53.368723 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:53.368727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:53.368728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:53.368811 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:53.369102 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:53.369105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:53.369107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:53.369333 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:53.369432 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:53.369857 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 18:37:53.370026 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:37:53.370970 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:53.357664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44769 (local address 127.0.98.2:58851)
0619 18:37:53.358130 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:53.358174 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:53.359521 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:53.360106 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:53.360126 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:53.360637 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:53.361257 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:53.361273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:53.362269 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:53.362273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:53.363613 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:53.363623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:53.364071 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:53.364075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:53.364266 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:53.364821 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:53.364840 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:53.366443 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:53.368368 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:53.368373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:53.368385 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:53.368627 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:53.368900 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:53.368902 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:53.368903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:53.369022 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:53.369426 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:53.369431 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:53.369663 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 18:37:53.369818 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 18:37:53.370115 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 18:37:53.372211 4314 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44769
I20260619 18:37:53.372447 4314 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:53.372947 4314 heartbeater.cc:511] Master 127.0.98.62:44769 requested a full tablet report, sending...
I20260619 18:37:53.374018 3989 ts_manager.cc:194] Registered new tserver with Master: 928c605f47aa46aca9f92c5a92654047 (127.0.98.2:35575)
I20260619 18:37:53.374732 3989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:37:53.386816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44769
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:53.495326 4319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:53.495604 4319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:53.495647 4319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:53.499673 4319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:53.499732 4319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:53.499810 4319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:37:53.504997 4319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:53.506043 4319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:53.507015 4319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:53.516287 4325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:53.516287 4327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:53.516279 4324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:53.516538 4319 server_base.cc:1034] running on GCE node
I20260619 18:37:53.517007 4319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:53.517827 4319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:53.518996 4319 hybrid_clock.cc:648] HybridClock initialized: now 1781894273518973 us; error 40 us; skew 500 ppm
Jun 19 18:37:53 dist-test-slave-rxv6 krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:53.521696 4319 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:37:53.522661 4319 webserver.cc:460] Webserver started at http://127.0.98.3:39975/ using document root <none> and password file <none>
I20260619 18:37:53.523238 4319 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:53.523299 4319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:53.523485 4319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:53.525617 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "6cdd5e2a4b43442ea823c91f3a19219d"
format_stamp: "Formatted at 2026-06-19 18:37:53 on dist-test-slave-rxv6"
server_key: "b3405e4f1211bf8c0a8dd36ada671a3d"
server_key_iv: "0a5f96adb5af1bc6ae524e65b75f248c"
server_key_version: "encryptionkey@0"
I20260619 18:37:53.526208 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6cdd5e2a4b43442ea823c91f3a19219d"
format_stamp: "Formatted at 2026-06-19 18:37:53 on dist-test-slave-rxv6"
server_key: "b3405e4f1211bf8c0a8dd36ada671a3d"
server_key_iv: "0a5f96adb5af1bc6ae524e65b75f248c"
server_key_version: "encryptionkey@0"
I20260619 18:37:53.529812 4319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:37:53.532207 4334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:53.533397 4319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:37:53.533494 4319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "6cdd5e2a4b43442ea823c91f3a19219d"
format_stamp: "Formatted at 2026-06-19 18:37:53 on dist-test-slave-rxv6"
server_key: "b3405e4f1211bf8c0a8dd36ada671a3d"
server_key_iv: "0a5f96adb5af1bc6ae524e65b75f248c"
server_key_version: "encryptionkey@0"
I20260619 18:37:53.533596 4319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:53.553285 4319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:53.556535 4319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:53.556720 4319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:53.557301 4319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:53.558138 4319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:53.558189 4319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:53.558233 4319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:53.558261 4319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:53.566704 4319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34899
I20260619 18:37:53.566735 4447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34899 every 8 connection(s)
I20260619 18:37:53.567647 4319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:37:53.572567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 4319
I20260619 18:37:53.572672 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:37:53.572934 392 external_mini_cluster.cc:1428] Setting key 996a7465383b95a620a7f940f04d3017
Jun 19 18:37:53 dist-test-slave-rxv6 krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:53.580850 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:53.569400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53091 (local address 127.0.98.62:44769)
0619 18:37:53.569566 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:53.569570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:53.570491 (+ 921us) server_negotiation.cc:407] Connection header received
0619 18:37:53.571822 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:53.571825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:53.571878 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:53.571955 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:53.573231 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:53.573936 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:53.575241 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:53.575467 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:53.575559 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:53.577818 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:53.577822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:53.577825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:53.577851 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:53.579002 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:53.579460 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:53.579464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:53.579466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:53.579540 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:53.579839 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:53.579842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:53.579844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:53.580018 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:53.580100 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:53.580569 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 18:37:53.580685 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:37:53.581619 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:53.569833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44769 (local address 127.0.98.3:53091)
0619 18:37:53.570343 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:53.570386 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:53.571606 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:53.572109 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:53.572116 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:53.572532 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:53.573086 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:53.573102 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:53.574040 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:53.574044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:53.575137 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:53.575146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:53.575605 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:53.575609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:53.575761 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:53.576289 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:53.576328 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:53.577659 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:53.579101 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:53.579107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:53.579122 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:53.579373 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:53.579617 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:53.579621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:53.579622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:53.579753 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:53.580107 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:53.580113 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:53.580356 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 18:37:53.580517 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 18:37:53.580832 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":120,"threads_started":1}
I20260619 18:37:53.582661 4448 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44769
I20260619 18:37:53.582882 4448 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:53.583365 4448 heartbeater.cc:511] Master 127.0.98.62:44769 requested a full tablet report, sending...
I20260619 18:37:53.584378 3989 ts_manager.cc:194] Registered new tserver with Master: 6cdd5e2a4b43442ea823c91f3a19219d (127.0.98.3:34899)
I20260619 18:37:53.584926 3989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53091
I20260619 18:37:53.586771 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 18:37:53 dist-test-slave-rxv6 krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:37:53 dist-test-slave-rxv6 krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 18:37:53.611249 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:53.602239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39362 (local address 127.0.98.1:36681)
0619 18:37:53.602591 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:53.602602 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:53.602624 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:37:53.602672 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:53.602677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:53.602886 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:53.603040 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:53.603809 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:53.604602 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:53.605438 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:53.605794 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:53.605872 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:53.608168 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:53.608176 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:53.608190 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:53.608212 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:53.609599 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:53.610005 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:53.610010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:53.610012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:53.610087 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:53.610321 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:53.610325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:53.610327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:53.610558 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:53.610748 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:53.610937 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 18:37:53.611084 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":154,"threads_started":1}
Jun 19 18:37:53 dist-test-slave-rxv6 krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:53.621551 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:53.613839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42928 (local address 127.0.98.62:44769)
0619 18:37:53.614015 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:53.614018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:53.614061 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:37:53.614165 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:53.614168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:53.614221 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:53.614287 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:53.614822 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:53.615353 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:53.616190 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:53.616405 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:53.616495 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:53.618777 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:53.618780 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:53.618782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:53.618806 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:53.620013 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:53.620417 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:53.620421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:53.620423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:53.620496 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:53.620723 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:53.620726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:53.620728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:53.620952 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:53.621071 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:53.621288 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 18:37:53.621401 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260619 18:37:53.621994 3989 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42928
I20260619 18:37:53.622545 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 4051
I20260619 18:37:53.627379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 4185
I20260619 18:37:53.632082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 4319
I20260619 18:37:53.636698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 3957
2026-06-19T18:37:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2016 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:37:53 dist-test-slave-rxv6 krb5kdc[4460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:37:53 dist-test-slave-rxv6 krb5kdc[4460](info): set up 2 sockets
Jun 19 18:37:53 dist-test-slave-rxv6 krb5kdc[4460](info): commencing operation
krb5kdc: starting...
W20260619 18:37:54.597724 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:37:54 dist-test-slave-rxv6 krb5kdc[4460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:37:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:37:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:37:54.717018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39115
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:54.824335 4473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:54.824610 4473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:54.824653 4473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:54.828634 4473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:37:54.828693 4473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:54.828711 4473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:54.828727 4473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:37:54.828740 4473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:37:54.833933 4473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:54.834916 4473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:54.836236 4473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:54.844336 4479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:54.844327 4481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:54.844412 4478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:54.844617 4473 server_base.cc:1034] running on GCE node
I20260619 18:37:54.845201 4473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:54.846390 4473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:54.847554 4473 hybrid_clock.cc:648] HybridClock initialized: now 1781894274847524 us; error 41 us; skew 500 ppm
Jun 19 18:37:54 dist-test-slave-rxv6 krb5kdc[4460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894274, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:54.850255 4473 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:37:54.851016 4473 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:54.851082 4473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:54.851256 4473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:54.853296 4473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "f3f5945941f44096aa7e6d58fc64cc9e"
format_stamp: "Formatted at 2026-06-19 18:37:54 on dist-test-slave-rxv6"
server_key: "d6be1f39bba56deb5ef6e5099facd7d6"
server_key_iv: "79b3e4590870cc30f80d5a7bbd44f3fa"
server_key_version: "encryptionkey@0"
I20260619 18:37:54.853824 4473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "f3f5945941f44096aa7e6d58fc64cc9e"
format_stamp: "Formatted at 2026-06-19 18:37:54 on dist-test-slave-rxv6"
server_key: "d6be1f39bba56deb5ef6e5099facd7d6"
server_key_iv: "79b3e4590870cc30f80d5a7bbd44f3fa"
server_key_version: "encryptionkey@0"
I20260619 18:37:54.857458 4473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:37:54.859968 4487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:54.861246 4473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:37:54.861423 4473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "f3f5945941f44096aa7e6d58fc64cc9e"
format_stamp: "Formatted at 2026-06-19 18:37:54 on dist-test-slave-rxv6"
server_key: "d6be1f39bba56deb5ef6e5099facd7d6"
server_key_iv: "79b3e4590870cc30f80d5a7bbd44f3fa"
server_key_version: "encryptionkey@0"
I20260619 18:37:54.861527 4473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:54.885332 4473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:54.888411 4473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:54.888588 4473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:54.894920 4473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39115
I20260619 18:37:54.895148 4539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39115 every 8 connection(s)
I20260619 18:37:54.895545 4473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:37:54.898192 4540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:54.902863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 4473
I20260619 18:37:54.902983 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:37:54.903223 392 external_mini_cluster.cc:1428] Setting key fc943513918f47c174dccf23b586fdfc
I20260619 18:37:54.903916 4540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e: Bootstrap starting.
I20260619 18:37:54.906196 4540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:54.907083 4540 log.cc:824] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:54.909159 4540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e: No bootstrap required, opened a new log
Jun 19 18:37:54 dist-test-slave-rxv6 krb5kdc[4460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:54.913251 4540 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f5945941f44096aa7e6d58fc64cc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39115 } }
I20260619 18:37:54.913479 4540 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:54.913561 4540 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f5945941f44096aa7e6d58fc64cc9e, State: Initialized, Role: FOLLOWER
I20260619 18:37:54.914023 4540 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f5945941f44096aa7e6d58fc64cc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39115 } }
I20260619 18:37:54.914186 4540 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:37:54.914250 4540 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:37:54.914403 4540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:54.915345 4540 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f5945941f44096aa7e6d58fc64cc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39115 } }
I20260619 18:37:54.915685 4540 leader_election.cc:302] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3f5945941f44096aa7e6d58fc64cc9e; no voters:
I20260619 18:37:54.916039 4540 leader_election.cc:288] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:37:54.916118 4545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:54.916364 4545 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [term 1 LEADER]: Becoming Leader. State: Replica: f3f5945941f44096aa7e6d58fc64cc9e, State: Running, Role: LEADER
I20260619 18:37:54.916721 4545 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f5945941f44096aa7e6d58fc64cc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39115 } }
I20260619 18:37:54.917433 4540 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:37:54.917856 4545 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3f5945941f44096aa7e6d58fc64cc9e. Latest consensus state: current_term: 1 leader_uuid: "f3f5945941f44096aa7e6d58fc64cc9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f5945941f44096aa7e6d58fc64cc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39115 } } }
I20260619 18:37:54.918047 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:54.904605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.0.98.62:39115)
0619 18:37:54.905007 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:54.905023 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:54.905057 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:37:54.905691 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:54.905709 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:54.906089 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:54.906412 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:54.907241 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:54.908357 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:54.909501 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:54.909992 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:54.910115 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:54.912785 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:54.912791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:54.912811 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:54.912837 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:54.914548 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:54.914947 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:54.914953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:54.914959 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:54.915070 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:54.915319 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:54.915323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:54.915324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:54.915766 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:54.915934 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:54.916724 (+ 790us) server_negotiation.cc:299] Negotiation successful
0619 18:37:54.917031 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":117,"threads_started":1}
I20260619 18:37:54.918073 4545 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:54.918252 4546 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3f5945941f44096aa7e6d58fc64cc9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3f5945941f44096aa7e6d58fc64cc9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39115 } } }
I20260619 18:37:54.918331 4546 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:54.918677 4553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:37:54.921715 4553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:37:54.927354 4553 catalog_manager.cc:1347] Generated new cluster ID: 3e9807fa3f9c4311a30ba462c9b38bae
I20260619 18:37:54.927429 4553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:37:54.939193 4553 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:37:54.940488 4553 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:37:54.953682 4553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e: Generated new TSK 0
I20260619 18:37:54.954635 4553 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:37:54.992686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39115
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:55.131728 4566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:55.131989 4566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:55.132033 4566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:55.136236 4566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:55.136317 4566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:55.136379 4566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:37:55.141660 4566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:55.142730 4566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:55.143735 4566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:55.152758 4574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:55.152771 4571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:55.152773 4572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:55.153005 4566 server_base.cc:1034] running on GCE node
I20260619 18:37:55.153446 4566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:55.154070 4566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:55.155253 4566 hybrid_clock.cc:648] HybridClock initialized: now 1781894275155221 us; error 41 us; skew 500 ppm
Jun 19 18:37:55 dist-test-slave-rxv6 krb5kdc[4460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:55.158042 4566 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:37:55.158795 4566 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:55.158862 4566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:55.159047 4566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:55.161053 4566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "357d2f489da04e8182b5da95aaf53c83"
format_stamp: "Formatted at 2026-06-19 18:37:55 on dist-test-slave-rxv6"
server_key: "972425f38cf6c8a6f877a3e68af5d5d4"
server_key_iv: "f62ce3272dee1c356406a1244745e7ee"
server_key_version: "encryptionkey@0"
I20260619 18:37:55.161584 4566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "357d2f489da04e8182b5da95aaf53c83"
format_stamp: "Formatted at 2026-06-19 18:37:55 on dist-test-slave-rxv6"
server_key: "972425f38cf6c8a6f877a3e68af5d5d4"
server_key_iv: "f62ce3272dee1c356406a1244745e7ee"
server_key_version: "encryptionkey@0"
I20260619 18:37:55.165215 4566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:37:55.167732 4580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:55.168907 4566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 18:37:55.169013 4566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "357d2f489da04e8182b5da95aaf53c83"
format_stamp: "Formatted at 2026-06-19 18:37:55 on dist-test-slave-rxv6"
server_key: "972425f38cf6c8a6f877a3e68af5d5d4"
server_key_iv: "f62ce3272dee1c356406a1244745e7ee"
server_key_version: "encryptionkey@0"
I20260619 18:37:55.169111 4566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:55.177973 4566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:55.180497 4566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:55.180665 4566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:55.181172 4566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:55.181967 4566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:55.182010 4566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:55.182060 4566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:55.182088 4566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:55.189975 4566 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40281
I20260619 18:37:55.189989 4693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40281 every 8 connection(s)
I20260619 18:37:55.190526 4566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:37:55.199478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 4566
I20260619 18:37:55.199630 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
Jun 19 18:37:55 dist-test-slave-rxv6 krb5kdc[4460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:55.199942 392 external_mini_cluster.cc:1428] Setting key bd0e0fd9a6dce28cd25d89cca0dffffe
I20260619 18:37:55.204008 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.192101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46761 (local address 127.0.98.62:39115)
0619 18:37:55.192250 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:55.192255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:55.193058 (+ 803us) server_negotiation.cc:407] Connection header received
0619 18:37:55.194356 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:55.194359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:55.194412 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:55.194486 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:55.195582 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.196102 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.197558 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.197786 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.197866 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.200444 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:55.200447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:55.200449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:55.200476 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:55.201945 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:55.202505 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:55.202507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:55.202509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:55.202577 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:55.202937 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:55.202940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:55.202942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:55.203160 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:55.203253 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:55.203723 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 18:37:55.203860 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:37:55.204756 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.192465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39115 (local address 127.0.98.1:46761)
0619 18:37:55.192899 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:55.192939 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:55.194161 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:55.194643 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:55.194649 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:55.194987 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:55.195466 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:55.195478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.196199 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.196201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:55.197442 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:55.197452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.197930 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.197934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:55.198093 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.198650 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:55.198665 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:55.200218 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:55.202071 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:55.202077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:55.202095 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:55.202422 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:55.202646 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:55.202648 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:55.202651 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:55.202836 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:55.203244 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:55.203249 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:55.203498 (+ 249us) client_negotiation.cc:769] Sending connection context
0619 18:37:55.203654 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 18:37:55.203910 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
I20260619 18:37:55.205976 4694 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39115
I20260619 18:37:55.206228 4694 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:55.206650 4694 heartbeater.cc:511] Master 127.0.98.62:39115 requested a full tablet report, sending...
I20260619 18:37:55.208199 4504 ts_manager.cc:194] Registered new tserver with Master: 357d2f489da04e8182b5da95aaf53c83 (127.0.98.1:40281)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 18:37:55.209502 4504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:37:55.224852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39115
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:55.334512 4699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:55.334791 4699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:55.334834 4699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:55.338917 4699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:55.338990 4699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:55.339051 4699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:37:55.344470 4699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:55.345537 4699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:55.346551 4699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:55.355520 4707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:55.355520 4705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:55.355546 4704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:55.355695 4699 server_base.cc:1034] running on GCE node
I20260619 18:37:55.356233 4699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:55.356914 4699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:55.358083 4699 hybrid_clock.cc:648] HybridClock initialized: now 1781894275358062 us; error 41 us; skew 500 ppm
Jun 19 18:37:55 dist-test-slave-rxv6 krb5kdc[4460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:55.360798 4699 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:37:55.361524 4699 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:55.361594 4699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:55.361778 4699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:55.363837 4699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "1fac03300e1748918702ae79bd465a90"
format_stamp: "Formatted at 2026-06-19 18:37:55 on dist-test-slave-rxv6"
server_key: "a6b95bfe0a923b61b7abf1dc81c2a8f3"
server_key_iv: "9fc37db3466fc85ba63834eb601eddb9"
server_key_version: "encryptionkey@0"
I20260619 18:37:55.364388 4699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1fac03300e1748918702ae79bd465a90"
format_stamp: "Formatted at 2026-06-19 18:37:55 on dist-test-slave-rxv6"
server_key: "a6b95bfe0a923b61b7abf1dc81c2a8f3"
server_key_iv: "9fc37db3466fc85ba63834eb601eddb9"
server_key_version: "encryptionkey@0"
I20260619 18:37:55.367990 4699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:37:55.370296 4713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:55.371543 4699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:37:55.371654 4699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "1fac03300e1748918702ae79bd465a90"
format_stamp: "Formatted at 2026-06-19 18:37:55 on dist-test-slave-rxv6"
server_key: "a6b95bfe0a923b61b7abf1dc81c2a8f3"
server_key_iv: "9fc37db3466fc85ba63834eb601eddb9"
server_key_version: "encryptionkey@0"
I20260619 18:37:55.371743 4699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:55.384370 4699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:55.387413 4699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:55.387588 4699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:55.388139 4699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:55.388934 4699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:55.388979 4699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:55.389031 4699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:55.389060 4699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:55.397326 4699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45693
I20260619 18:37:55.397341 4826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45693 every 8 connection(s)
I20260619 18:37:55.397933 4699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:37:55.401058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 4699
I20260619 18:37:55.401171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:37:55.401432 392 external_mini_cluster.cc:1428] Setting key 8c9371d420b8114b9d81dbf6abe882d9
Jun 19 18:37:55 dist-test-slave-rxv6 krb5kdc[4460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 18:37:55.411412 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.399718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56403 (local address 127.0.98.62:39115)
0619 18:37:55.399911 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:55.399916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:55.400769 (+ 853us) server_negotiation.cc:407] Connection header received
0619 18:37:55.402196 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:55.402199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:55.402245 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:55.402322 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:55.403655 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.404166 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.405255 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.405464 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.405548 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.408121 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:55.408124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:55.408127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:55.408149 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:55.409416 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:55.410014 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:55.410016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:55.410018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:55.410093 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:55.410418 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:55.410420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:55.410421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:55.410608 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:55.410678 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:55.411182 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 18:37:55.411282 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":71}
I20260619 18:37:55.412240 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.400142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39115 (local address 127.0.98.2:56403)
0619 18:37:55.400618 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:55.400660 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:55.401968 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:55.402525 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:55.402543 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:55.403018 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:55.403512 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:55.403526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.404276 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.404279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:55.405147 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:55.405155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.405596 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.405599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:55.405755 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.406294 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:55.406309 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:55.407959 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:55.409527 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:55.409533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:55.409548 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:55.409868 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:55.410195 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:55.410197 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:55.410198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:55.410336 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:55.410716 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:55.410723 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:55.410966 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 18:37:55.411117 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 18:37:55.411365 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 18:37:55.413335 4827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39115
I20260619 18:37:55.413563 4827 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:55.414098 4827 heartbeater.cc:511] Master 127.0.98.62:39115 requested a full tablet report, sending...
I20260619 18:37:55.415136 4504 ts_manager.cc:194] Registered new tserver with Master: 1fac03300e1748918702ae79bd465a90 (127.0.98.2:45693)
I20260619 18:37:55.415693 4504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:37:55.427621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39115
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:55.534957 4832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:55.535198 4832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:55.535240 4832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:55.539136 4832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:55.539192 4832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:55.539249 4832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:37:55.544272 4832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:55.545207 4832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:55.546180 4832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:55.554962 4838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:55.555045 4840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:55.554966 4837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:55.555234 4832 server_base.cc:1034] running on GCE node
I20260619 18:37:55.555733 4832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:55.556375 4832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:55.557538 4832 hybrid_clock.cc:648] HybridClock initialized: now 1781894275557518 us; error 33 us; skew 500 ppm
Jun 19 18:37:55 dist-test-slave-rxv6 krb5kdc[4460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:55.560209 4832 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:37:55.560894 4832 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:55.560957 4832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:55.561132 4832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:55.563093 4832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "4baadf245026431fb608d41c78cee2b2"
format_stamp: "Formatted at 2026-06-19 18:37:55 on dist-test-slave-rxv6"
server_key: "2e2f1ff38297a1a419f41f8e182b7c63"
server_key_iv: "cdf6356669433c5f75a3b7f2ef02d423"
server_key_version: "encryptionkey@0"
I20260619 18:37:55.563609 4832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4baadf245026431fb608d41c78cee2b2"
format_stamp: "Formatted at 2026-06-19 18:37:55 on dist-test-slave-rxv6"
server_key: "2e2f1ff38297a1a419f41f8e182b7c63"
server_key_iv: "cdf6356669433c5f75a3b7f2ef02d423"
server_key_version: "encryptionkey@0"
I20260619 18:37:55.567112 4832 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:37:55.569396 4846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:55.570641 4832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:37:55.570747 4832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "4baadf245026431fb608d41c78cee2b2"
format_stamp: "Formatted at 2026-06-19 18:37:55 on dist-test-slave-rxv6"
server_key: "2e2f1ff38297a1a419f41f8e182b7c63"
server_key_iv: "cdf6356669433c5f75a3b7f2ef02d423"
server_key_version: "encryptionkey@0"
I20260619 18:37:55.570837 4832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:55.595904 4832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:55.599063 4832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:55.599241 4832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:55.599805 4832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:55.600644 4832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:55.600692 4832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:55.600744 4832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:55.600771 4832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:55.608922 4832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38777
I20260619 18:37:55.608945 4959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38777 every 8 connection(s)
I20260619 18:37:55.609467 4832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:37:55.613657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 4832
I20260619 18:37:55.613781 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:37:55.614114 392 external_mini_cluster.cc:1428] Setting key 040535d9a8bd8b8e33de35a432015649
Jun 19 18:37:55 dist-test-slave-rxv6 krb5kdc[4460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:55.622351 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.611154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47769 (local address 127.0.98.62:39115)
0619 18:37:55.611308 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:55.611313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:55.612101 (+ 788us) server_negotiation.cc:407] Connection header received
0619 18:37:55.613391 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:55.613394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:55.613479 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:55.613553 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:55.615014 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.615624 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.616830 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.617049 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.617136 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.619382 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:55.619385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:55.619387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:55.619419 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:55.620521 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:55.620957 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:55.620961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:55.620963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:55.621036 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:55.621321 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:55.621324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:55.621326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:55.621499 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:55.621583 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:55.622074 (+ 491us) server_negotiation.cc:299] Negotiation successful
0619 18:37:55.622192 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:37:55.623134 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.611533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39115 (local address 127.0.98.3:47769)
0619 18:37:55.611955 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:55.611996 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:55.613189 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:55.613760 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:55.613770 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:55.614266 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:55.614860 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:55.614877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.615743 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.615747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:55.616715 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:55.616725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.617183 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.617187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:55.617344 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.617844 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:55.617859 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:55.619238 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:55.620622 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:55.620627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:55.620642 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:55.620869 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:55.621113 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:55.621116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:55.621118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:55.621240 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:55.621592 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:55.621598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:55.621821 (+ 223us) client_negotiation.cc:769] Sending connection context
0619 18:37:55.622014 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 18:37:55.622350 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":108,"threads_started":1}
I20260619 18:37:55.624166 4960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39115
I20260619 18:37:55.624372 4960 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:55.624795 4960 heartbeater.cc:511] Master 127.0.98.62:39115 requested a full tablet report, sending...
I20260619 18:37:55.625751 4504 ts_manager.cc:194] Registered new tserver with Master: 4baadf245026431fb608d41c78cee2b2 (127.0.98.3:38777)
I20260619 18:37:55.626318 4504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47769
I20260619 18:37:55.627655 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:37:55.635928 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.629514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.0.98.62:39115)
0619 18:37:55.629617 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:55.629620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:55.629798 (+ 178us) server_negotiation.cc:407] Connection header received
0619 18:37:55.629895 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:55.629897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:55.629957 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:55.630015 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:55.630591 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.631068 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.631809 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.632018 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.632091 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.633232 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:55.633235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:55.633237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:55.633256 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:55.634570 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:55.634943 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:55.634946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:55.634947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:55.635028 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:55.635223 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:55.635225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:55.635226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:55.635476 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:55.635581 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:55.635720 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 18:37:55.635806 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 18:37:55.638468 4504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43428:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:37:55.641309 4504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:37:55.655252 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.649159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38777 (local address 127.0.0.1:44146)
0619 18:37:55.649936 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:55.649988 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:55.650111 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:55.650877 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:55.650883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:55.650905 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:55.651033 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:55.651039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.652527 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.652530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:55.653699 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:55.653707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.654714 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.654717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:55.654827 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.654862 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 18:37:55.654970 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 18:37:55.655086 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":104,"thread_start_us":71,"threads_started":1}
I20260619 18:37:55.655263 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.649303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.0.98.3:38777)
0619 18:37:55.650383 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:55.650395 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:55.650418 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:37:55.650493 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:55.650498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:55.650714 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:55.650875 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:55.651170 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.652381 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.653805 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.654588 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.654692 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.654838 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:55.655008 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 18:37:55.655111 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":978,"thread_start_us":97,"threads_started":1}
I20260619 18:37:55.655649 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.649808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.0.98.2:45693)
0619 18:37:55.650434 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:55.650444 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:55.650464 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:37:55.650534 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:55.650540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:55.650748 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:55.650891 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:55.651242 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.652367 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.654167 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.654703 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.654801 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.654946 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:55.655398 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 18:37:55.655488 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":118,"threads_started":1}
I20260619 18:37:55.656296 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.649300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45693 (local address 127.0.0.1:44100)
0619 18:37:55.650047 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:55.650058 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:55.650120 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:55.651036 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:55.651038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:55.651052 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:55.651133 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:55.651139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.652483 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.652486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:55.654070 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:55.654077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.655003 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.655007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:55.655140 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.655177 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 18:37:55.655238 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 18:37:55.655299 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":139,"thread_start_us":99,"threads_started":1}
I20260619 18:37:55.656697 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.650011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41934 (local address 127.0.98.1:40281)
0619 18:37:55.650744 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:55.650754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:55.650776 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:37:55.650841 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:55.650846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:55.651005 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:55.651135 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:55.651334 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.652273 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.653998 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.654750 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.654835 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.654938 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:55.656448 (+ 1510us) server_negotiation.cc:299] Negotiation successful
0619 18:37:55.656545 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":142,"threads_started":1}
I20260619 18:37:55.656920 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.649511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40281 (local address 127.0.0.1:41934)
0619 18:37:55.650377 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:55.650390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:55.650473 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:55.651135 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:55.651138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:55.651155 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:55.651248 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:55.651255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.652386 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.652390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:55.653867 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:55.653876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.654997 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.655001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:55.655143 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.655187 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 18:37:55.655253 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 18:37:55.655321 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":167,"thread_start_us":101,"threads_started":1}
I20260619 18:37:55.658244 4894 tablet_service.cc:1467] Processing CreateTablet for tablet 6b2ccad5fc1d48fc8c98055c9a857ec0 (DEFAULT_TABLE table=test-table [id=6c7df0695a294fd0820ae98e462f58db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:55.658672 4761 tablet_service.cc:1467] Processing CreateTablet for tablet 6b2ccad5fc1d48fc8c98055c9a857ec0 (DEFAULT_TABLE table=test-table [id=6c7df0695a294fd0820ae98e462f58db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:55.659209 4894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b2ccad5fc1d48fc8c98055c9a857ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:55.659523 4761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b2ccad5fc1d48fc8c98055c9a857ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:55.664944 4977 tablet_bootstrap.cc:492] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2: Bootstrap starting.
I20260619 18:37:55.665362 4978 tablet_bootstrap.cc:492] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90: Bootstrap starting.
I20260619 18:37:55.667632 4978 tablet_bootstrap.cc:654] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:55.667488 4628 tablet_service.cc:1467] Processing CreateTablet for tablet 6b2ccad5fc1d48fc8c98055c9a857ec0 (DEFAULT_TABLE table=test-table [id=6c7df0695a294fd0820ae98e462f58db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:55.668545 4628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b2ccad5fc1d48fc8c98055c9a857ec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:55.668601 4977 tablet_bootstrap.cc:654] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:55.668673 4978 log.cc:824] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:55.669721 4977 log.cc:824] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:55.670435 4978 tablet_bootstrap.cc:492] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90: No bootstrap required, opened a new log
I20260619 18:37:55.670617 4978 ts_tablet_manager.cc:1397] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:37:55.671650 4977 tablet_bootstrap.cc:492] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2: No bootstrap required, opened a new log
I20260619 18:37:55.671816 4977 ts_tablet_manager.cc:1397] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 18:37:55.673815 4978 raft_consensus.cc:348] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4baadf245026431fb608d41c78cee2b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38777 } } peers { permanent_uuid: "1fac03300e1748918702ae79bd465a90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "357d2f489da04e8182b5da95aaf53c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40281 } }
I20260619 18:37:55.674077 4978 raft_consensus.cc:374] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:55.674129 4978 raft_consensus.cc:729] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fac03300e1748918702ae79bd465a90, State: Initialized, Role: FOLLOWER
I20260619 18:37:55.674588 4978 consensus_queue.cc:260] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4baadf245026431fb608d41c78cee2b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38777 } } peers { permanent_uuid: "1fac03300e1748918702ae79bd465a90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "357d2f489da04e8182b5da95aaf53c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40281 } }
I20260619 18:37:55.675251 4827 heartbeater.cc:503] Master 127.0.98.62:39115 was elected leader, sending a full tablet report...
I20260619 18:37:55.675591 4978 ts_tablet_manager.cc:1428] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 18:37:55.675712 4982 tablet_bootstrap.cc:492] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83: Bootstrap starting.
I20260619 18:37:55.676731 4977 raft_consensus.cc:348] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4baadf245026431fb608d41c78cee2b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38777 } } peers { permanent_uuid: "1fac03300e1748918702ae79bd465a90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "357d2f489da04e8182b5da95aaf53c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40281 } }
I20260619 18:37:55.677009 4977 raft_consensus.cc:374] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:55.677069 4977 raft_consensus.cc:729] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4baadf245026431fb608d41c78cee2b2, State: Initialized, Role: FOLLOWER
I20260619 18:37:55.677491 4977 consensus_queue.cc:260] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4baadf245026431fb608d41c78cee2b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38777 } } peers { permanent_uuid: "1fac03300e1748918702ae79bd465a90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "357d2f489da04e8182b5da95aaf53c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40281 } }
I20260619 18:37:55.678117 4960 heartbeater.cc:503] Master 127.0.98.62:39115 was elected leader, sending a full tablet report...
I20260619 18:37:55.678369 4982 tablet_bootstrap.cc:654] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:55.678433 4977 ts_tablet_manager.cc:1428] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:37:55.679831 4982 log.cc:824] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:55.681964 4982 tablet_bootstrap.cc:492] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83: No bootstrap required, opened a new log
I20260619 18:37:55.682147 4982 ts_tablet_manager.cc:1397] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 18:37:55.684827 4982 raft_consensus.cc:348] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4baadf245026431fb608d41c78cee2b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38777 } } peers { permanent_uuid: "1fac03300e1748918702ae79bd465a90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "357d2f489da04e8182b5da95aaf53c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40281 } }
I20260619 18:37:55.685035 4982 raft_consensus.cc:374] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:55.685088 4982 raft_consensus.cc:729] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357d2f489da04e8182b5da95aaf53c83, State: Initialized, Role: FOLLOWER
I20260619 18:37:55.685453 4982 consensus_queue.cc:260] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4baadf245026431fb608d41c78cee2b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38777 } } peers { permanent_uuid: "1fac03300e1748918702ae79bd465a90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "357d2f489da04e8182b5da95aaf53c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40281 } }
I20260619 18:37:55.686013 4694 heartbeater.cc:503] Master 127.0.98.62:39115 was elected leader, sending a full tablet report...
I20260619 18:37:55.686265 4982 ts_tablet_manager.cc:1428] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260619 18:37:55.691618 4695 tablet.cc:2369] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:37:55.789058 4985 raft_consensus.cc:482] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:37:55.789289 4985 raft_consensus.cc:504] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4baadf245026431fb608d41c78cee2b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38777 } } peers { permanent_uuid: "1fac03300e1748918702ae79bd465a90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "357d2f489da04e8182b5da95aaf53c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40281 } }
I20260619 18:37:55.790375 4985 leader_election.cc:288] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4baadf245026431fb608d41c78cee2b2 (127.0.98.3:38777), 1fac03300e1748918702ae79bd465a90 (127.0.98.2:45693)
I20260619 18:37:55.795346 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.790773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46457 (local address 127.0.98.3:38777)
0619 18:37:55.790903 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:55.790908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:55.791288 (+ 380us) server_negotiation.cc:407] Connection header received
0619 18:37:55.791388 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:55.791391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:55.791491 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:55.791583 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:55.791827 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.792316 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.794030 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.794692 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.794799 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.794866 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:55.795152 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 18:37:55.795215 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:37:55.795396 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.790666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38777 (local address 127.0.98.1:46457)
0619 18:37:55.791173 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:55.791188 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:55.791289 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:55.791606 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:55.791608 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:55.791622 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:55.791726 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:55.791733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.792421 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.792423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:55.793821 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:55.793831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.794879 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.794882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:55.795016 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.795045 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 18:37:55.795127 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 18:37:55.795211 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":101,"thread_start_us":99,"threads_started":1}
I20260619 18:37:55.796020 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.790857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33365 (local address 127.0.98.2:45693)
0619 18:37:55.790963 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:55.790967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:55.791139 (+ 172us) server_negotiation.cc:407] Connection header received
0619 18:37:55.791267 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:55.791271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:55.791321 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:55.791388 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:37:55.791708 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.792259 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.794075 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.794692 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.794798 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.794866 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:55.795832 (+ 966us) server_negotiation.cc:299] Negotiation successful
0619 18:37:55.795912 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:37:55.796032 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.790666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45693 (local address 127.0.98.1:33365)
0619 18:37:55.791031 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:55.791044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:55.791147 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:55.791430 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:55.791433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:55.791450 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:37:55.791593 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:55.791600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.792365 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.792369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:55.793851 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:55.793867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.794825 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.795595 (+ 770us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:55.795702 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.795727 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:37:55.795801 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 18:37:55.795870 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":54,"thread_start_us":107,"threads_started":1}
I20260619 18:37:55.796051 4914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2ccad5fc1d48fc8c98055c9a857ec0" candidate_uuid: "357d2f489da04e8182b5da95aaf53c83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4baadf245026431fb608d41c78cee2b2" is_pre_election: true
I20260619 18:37:55.796375 4914 raft_consensus.cc:2454] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 357d2f489da04e8182b5da95aaf53c83 in term 0.
I20260619 18:37:55.797019 4781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2ccad5fc1d48fc8c98055c9a857ec0" candidate_uuid: "357d2f489da04e8182b5da95aaf53c83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fac03300e1748918702ae79bd465a90" is_pre_election: true
I20260619 18:37:55.797048 4581 leader_election.cc:302] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 357d2f489da04e8182b5da95aaf53c83, 4baadf245026431fb608d41c78cee2b2; no voters:
I20260619 18:37:55.797317 4781 raft_consensus.cc:2454] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 357d2f489da04e8182b5da95aaf53c83 in term 0.
I20260619 18:37:55.797320 4985 raft_consensus.cc:2781] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:37:55.797408 4985 raft_consensus.cc:482] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:37:55.797451 4985 raft_consensus.cc:3037] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:55.798748 4985 raft_consensus.cc:504] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4baadf245026431fb608d41c78cee2b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38777 } } peers { permanent_uuid: "1fac03300e1748918702ae79bd465a90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "357d2f489da04e8182b5da95aaf53c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40281 } }
I20260619 18:37:55.799113 4985 leader_election.cc:288] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [CANDIDATE]: Term 1 election: Requested vote from peers 4baadf245026431fb608d41c78cee2b2 (127.0.98.3:38777), 1fac03300e1748918702ae79bd465a90 (127.0.98.2:45693)
I20260619 18:37:55.799485 4914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2ccad5fc1d48fc8c98055c9a857ec0" candidate_uuid: "357d2f489da04e8182b5da95aaf53c83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4baadf245026431fb608d41c78cee2b2"
I20260619 18:37:55.799619 4914 raft_consensus.cc:3037] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:55.799757 4781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b2ccad5fc1d48fc8c98055c9a857ec0" candidate_uuid: "357d2f489da04e8182b5da95aaf53c83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fac03300e1748918702ae79bd465a90"
I20260619 18:37:55.799892 4781 raft_consensus.cc:3037] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:55.800666 4914 raft_consensus.cc:2454] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357d2f489da04e8182b5da95aaf53c83 in term 1.
I20260619 18:37:55.800909 4781 raft_consensus.cc:2454] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 357d2f489da04e8182b5da95aaf53c83 in term 1.
I20260619 18:37:55.801237 4582 leader_election.cc:302] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1fac03300e1748918702ae79bd465a90, 357d2f489da04e8182b5da95aaf53c83, 4baadf245026431fb608d41c78cee2b2; no voters:
I20260619 18:37:55.801425 4985 raft_consensus.cc:2781] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:55.801656 4985 raft_consensus.cc:686] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [term 1 LEADER]: Becoming Leader. State: Replica: 357d2f489da04e8182b5da95aaf53c83, State: Running, Role: LEADER
I20260619 18:37:55.802031 4985 consensus_queue.cc:237] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4baadf245026431fb608d41c78cee2b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38777 } } peers { permanent_uuid: "1fac03300e1748918702ae79bd465a90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } } peers { permanent_uuid: "357d2f489da04e8182b5da95aaf53c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40281 } }
I20260619 18:37:55.804728 4501 catalog_manager.cc:5495] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 357d2f489da04e8182b5da95aaf53c83 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "357d2f489da04e8182b5da95aaf53c83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4baadf245026431fb608d41c78cee2b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fac03300e1748918702ae79bd465a90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "357d2f489da04e8182b5da95aaf53c83" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40281 } health_report { overall_health: HEALTHY } } }
I20260619 18:37:55.846599 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.843121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.0.98.1:40281)
0619 18:37:55.843246 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:55.843250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:55.843264 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:37:55.843384 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:55.843433 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:55.843496 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:55.843592 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:37:55.843836 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.844323 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.845165 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:55.845376 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:55.845472 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:55.845769 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:37:55.846229 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:37:55.846336 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:55.846409 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 18:37:55.846464 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:37:55.852082 4781 raft_consensus.cc:1261] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90 [term 1 FOLLOWER]: Refusing update from remote peer 357d2f489da04e8182b5da95aaf53c83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:55.852082 4914 raft_consensus.cc:1261] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2 [term 1 FOLLOWER]: Refusing update from remote peer 357d2f489da04e8182b5da95aaf53c83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:55.852782 4988 consensus_queue.cc:1035] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fac03300e1748918702ae79bd465a90" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:55.852942 4985 consensus_queue.cc:1035] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 357d2f489da04e8182b5da95aaf53c83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4baadf245026431fb608d41c78cee2b2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:55.860037 4993 mvcc.cc:205] Tried to move back new op lower bound from 7298638953883496448 to 7298638953687453696. Current Snapshot: MvccSnapshot[applied={T|T < 7298638953883496448}]
I20260619 18:37:55.867184 4503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:37:55.867340 4503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43428:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:37:55.869405 4503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3f5945941f44096aa7e6d58fc64cc9e: Sending DeleteTablet for 3 replicas of tablet 6b2ccad5fc1d48fc8c98055c9a857ec0
I20260619 18:37:55.870201 4628 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b2ccad5fc1d48fc8c98055c9a857ec0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41934
I20260619 18:37:55.870270 4894 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b2ccad5fc1d48fc8c98055c9a857ec0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44146
I20260619 18:37:55.870298 4761 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b2ccad5fc1d48fc8c98055c9a857ec0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44100
I20260619 18:37:55.870815 5003 tablet_replica.cc:330] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90: stopping tablet replica
I20260619 18:37:55.871066 5003 raft_consensus.cc:2229] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:37:55.871328 5003 raft_consensus.cc:2258] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:55.871378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 4566
I20260619 18:37:55.872917 5003 ts_tablet_manager.cc:1905] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:55.873801 5002 tablet_replica.cc:330] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2: stopping tablet replica
I20260619 18:37:55.874092 5002 raft_consensus.cc:2229] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:37:55.874358 5002 raft_consensus.cc:2258] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:55.875329 5003 ts_tablet_manager.cc:1918] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:37:55.875422 5003 log.cc:1196] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/6b2ccad5fc1d48fc8c98055c9a857ec0
I20260619 18:37:55.875694 5003 ts_tablet_manager.cc:1939] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 1fac03300e1748918702ae79bd465a90: Deleting consensus metadata
I20260619 18:37:55.875774 5002 ts_tablet_manager.cc:1905] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:55.876781 4489 catalog_manager.cc:4841] TS 1fac03300e1748918702ae79bd465a90 (127.0.98.2:45693): tablet 6b2ccad5fc1d48fc8c98055c9a857ec0 (table test-table [id=6c7df0695a294fd0820ae98e462f58db]) successfully deleted
I20260619 18:37:55.878218 5002 ts_tablet_manager.cc:1918] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:37:55.878309 5002 log.cc:1196] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/6b2ccad5fc1d48fc8c98055c9a857ec0
I20260619 18:37:55.878607 5002 ts_tablet_manager.cc:1939] T 6b2ccad5fc1d48fc8c98055c9a857ec0 P 4baadf245026431fb608d41c78cee2b2: Deleting consensus metadata
W20260619 18:37:55.879487 4491 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40281 (error 108)
I20260619 18:37:55.879664 4488 catalog_manager.cc:4841] TS 4baadf245026431fb608d41c78cee2b2 (127.0.98.3:38777): tablet 6b2ccad5fc1d48fc8c98055c9a857ec0 (table test-table [id=6c7df0695a294fd0820ae98e462f58db]) successfully deleted
I20260619 18:37:55.880224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 4699
I20260619 18:37:55.880465 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:55.880190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40281 (local address 127.0.0.1:41952)
0619 18:37:55.880317 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:55.880384 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 18:37:55.882270 4491 catalog_manager.cc:4584] TS 357d2f489da04e8182b5da95aaf53c83 (127.0.98.1:40281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b2ccad5fc1d48fc8c98055c9a857ec0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40281: connect: Connection refused (error 111)
I20260619 18:37:55.885403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 4832
I20260619 18:37:55.889809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 4473
2026-06-19T18:37:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2253 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T18:37:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:37:55Z Disabled control of system clock
I20260619 18:37:55.917052 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38411
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 18:37:56.023633 5008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:56.023909 5008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:56.023947 5008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:56.027966 5008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:37:56.028034 5008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:56.028048 5008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:56.028064 5008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:37:56.028077 5008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:37:56.032835 5008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38411
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:56.033836 5008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:56.035193 5008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:56.043238 5014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:56.043286 5016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:56.043234 5013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:56.043630 5008 server_base.cc:1034] running on GCE node
I20260619 18:37:56.044154 5008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:56.045233 5008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:56.046402 5008 hybrid_clock.cc:648] HybridClock initialized: now 1781894276046357 us; error 63 us; skew 500 ppm
I20260619 18:37:56.048221 5008 webserver.cc:460] Webserver started at http://127.0.98.62:35959/ using document root <none> and password file <none>
I20260619 18:37:56.048755 5008 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:56.048817 5008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:56.048990 5008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:56.050966 5008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "e00f3210bf4842b9b8d2e259c493a933"
format_stamp: "Formatted at 2026-06-19 18:37:56 on dist-test-slave-rxv6"
server_key: "4f0904feaee7bd29cfa4d830161aebbd"
server_key_iv: "1af488f5197d25018016ee4423c6e30e"
server_key_version: "encryptionkey@0"
I20260619 18:37:56.051466 5008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "e00f3210bf4842b9b8d2e259c493a933"
format_stamp: "Formatted at 2026-06-19 18:37:56 on dist-test-slave-rxv6"
server_key: "4f0904feaee7bd29cfa4d830161aebbd"
server_key_iv: "1af488f5197d25018016ee4423c6e30e"
server_key_version: "encryptionkey@0"
I20260619 18:37:56.054908 5008 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:37:56.057288 5022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:56.058571 5008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:37:56.058689 5008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "e00f3210bf4842b9b8d2e259c493a933"
format_stamp: "Formatted at 2026-06-19 18:37:56 on dist-test-slave-rxv6"
server_key: "4f0904feaee7bd29cfa4d830161aebbd"
server_key_iv: "1af488f5197d25018016ee4423c6e30e"
server_key_version: "encryptionkey@0"
I20260619 18:37:56.058787 5008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:56.063167 5008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:56.063745 5008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:56.063894 5008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:56.070459 5008 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38411
I20260619 18:37:56.070546 5074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38411 every 8 connection(s)
I20260619 18:37:56.071399 5008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:37:56.072206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 5008
I20260619 18:37:56.072302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:37:56.072546 392 external_mini_cluster.cc:1428] Setting key 65232ed484cd9703e58ef21a3c30c197
I20260619 18:37:56.074542 5075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:56.078481 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.073896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47444 (local address 127.0.98.62:38411)
0619 18:37:56.075078 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:56.075092 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:56.075127 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:37:56.075762 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:56.075779 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:56.076140 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:56.076453 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:56.077242 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:56.077246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:56.077257 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:37:56.077267 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:56.077321 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:37:56.077337 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:56.077461 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:56.077531 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 18:37:56.077776 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":657,"threads_started":1}
I20260619 18:37:56.080379 5075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933: Bootstrap starting.
I20260619 18:37:56.082515 5075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:56.083218 5075 log.cc:824] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:56.085085 5075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933: No bootstrap required, opened a new log
I20260619 18:37:56.089552 5075 raft_consensus.cc:348] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00f3210bf4842b9b8d2e259c493a933" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38411 } }
I20260619 18:37:56.089789 5075 raft_consensus.cc:374] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:56.089849 5075 raft_consensus.cc:729] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e00f3210bf4842b9b8d2e259c493a933, State: Initialized, Role: FOLLOWER
I20260619 18:37:56.090308 5075 consensus_queue.cc:260] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00f3210bf4842b9b8d2e259c493a933" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38411 } }
I20260619 18:37:56.090457 5075 raft_consensus.cc:388] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:37:56.090531 5075 raft_consensus.cc:482] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:37:56.090631 5075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:56.091506 5075 raft_consensus.cc:504] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00f3210bf4842b9b8d2e259c493a933" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38411 } }
I20260619 18:37:56.091799 5075 leader_election.cc:302] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e00f3210bf4842b9b8d2e259c493a933; no voters:
I20260619 18:37:56.092057 5075 leader_election.cc:288] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:37:56.092151 5080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:56.092360 5080 raft_consensus.cc:686] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [term 1 LEADER]: Becoming Leader. State: Replica: e00f3210bf4842b9b8d2e259c493a933, State: Running, Role: LEADER
I20260619 18:37:56.092656 5080 consensus_queue.cc:237] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00f3210bf4842b9b8d2e259c493a933" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38411 } }
I20260619 18:37:56.093180 5075 sys_catalog.cc:564] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:37:56.094143 5081 sys_catalog.cc:455] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e00f3210bf4842b9b8d2e259c493a933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00f3210bf4842b9b8d2e259c493a933" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38411 } } }
I20260619 18:37:56.094264 5081 sys_catalog.cc:458] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:56.094538 5082 sys_catalog.cc:455] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e00f3210bf4842b9b8d2e259c493a933. Latest consensus state: current_term: 1 leader_uuid: "e00f3210bf4842b9b8d2e259c493a933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00f3210bf4842b9b8d2e259c493a933" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38411 } } }
I20260619 18:37:56.094625 5082 sys_catalog.cc:458] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933 [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:56.094712 5089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:37:56.097690 5089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:37:56.102991 5089 catalog_manager.cc:1347] Generated new cluster ID: f1e2f32825fe47859d1f205d995e646e
I20260619 18:37:56.103070 5089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:37:56.117257 5089 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:37:56.117590 5089 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:37:56.124418 5089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933: Generated new TSK 0
I20260619 18:37:56.125113 5089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:37:56.136521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38411
--builtin_ntp_servers=127.0.98.20:38457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 18:37:56.262377 5099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:56.262632 5099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:56.262672 5099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:56.266727 5099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:56.266783 5099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:56.266839 5099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:37:56.271479 5099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:56.272578 5099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:56.273950 5099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:56.283002 5105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:56.283025 5104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:56.283164 5099 server_base.cc:1034] running on GCE node
W20260619 18:37:56.283079 5107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:56.283692 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:56.284286 5099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:56.285446 5099 hybrid_clock.cc:648] HybridClock initialized: now 1781894276285429 us; error 26 us; skew 500 ppm
I20260619 18:37:56.287237 5099 webserver.cc:460] Webserver started at http://127.0.98.1:34293/ using document root <none> and password file <none>
I20260619 18:37:56.287765 5099 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:56.287822 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:56.288008 5099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:56.290019 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "c7986d7213914077bab6744a9e26b80d"
format_stamp: "Formatted at 2026-06-19 18:37:56 on dist-test-slave-rxv6"
server_key: "09b86d447e5463feed1916ba0b236507"
server_key_iv: "1889a46c2e57a4459081044542772612"
server_key_version: "encryptionkey@0"
I20260619 18:37:56.290545 5099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c7986d7213914077bab6744a9e26b80d"
format_stamp: "Formatted at 2026-06-19 18:37:56 on dist-test-slave-rxv6"
server_key: "09b86d447e5463feed1916ba0b236507"
server_key_iv: "1889a46c2e57a4459081044542772612"
server_key_version: "encryptionkey@0"
I20260619 18:37:56.294138 5099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:37:56.296722 5113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:56.297889 5099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:37:56.298043 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "c7986d7213914077bab6744a9e26b80d"
format_stamp: "Formatted at 2026-06-19 18:37:56 on dist-test-slave-rxv6"
server_key: "09b86d447e5463feed1916ba0b236507"
server_key_iv: "1889a46c2e57a4459081044542772612"
server_key_version: "encryptionkey@0"
I20260619 18:37:56.298151 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:56.302584 5099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:56.303135 5099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:56.303277 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:56.303788 5099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:56.304649 5099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:56.304693 5099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:56.304734 5099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:56.304762 5099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:56.312564 5099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43721
I20260619 18:37:56.312582 5226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43721 every 8 connection(s)
I20260619 18:37:56.313409 5099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:37:56.319114 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.315052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36781 (local address 127.0.98.62:38411)
0619 18:37:56.315167 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:56.315171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:56.316354 (+ 1183us) server_negotiation.cc:407] Connection header received
0619 18:37:56.317572 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:56.317575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:56.317619 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:56.317713 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:56.318709 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:56.318713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:56.318716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:37:56.318728 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:56.318748 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:37:56.318756 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:56.318830 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:56.318943 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 18:37:56.318983 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 18:37:56.319926 5229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.315590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38411 (local address 127.0.98.1:36781)
0619 18:37:56.316219 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:56.316257 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:56.317381 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:56.317870 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:56.317877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:56.317999 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:56.318472 (+ 473us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:37:56.318474 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:56.318511 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:37:56.318574 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:56.318822 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:56.318829 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:56.318835 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:37:56.318977 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 18:37:56.319216 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":113,"threads_started":1}
I20260619 18:37:56.321002 5227 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38411
I20260619 18:37:56.321205 5227 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:56.321558 5227 heartbeater.cc:511] Master 127.0.98.62:38411 requested a full tablet report, sending...
I20260619 18:37:56.322959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 5099
I20260619 18:37:56.322969 5039 ts_manager.cc:194] Registered new tserver with Master: c7986d7213914077bab6744a9e26b80d (127.0.98.1:43721)
I20260619 18:37:56.323056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:37:56.323307 392 external_mini_cluster.cc:1428] Setting key 2392476e547e49d4c7333c9021094f2d
I20260619 18:37:56.325181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38411
--builtin_ntp_servers=127.0.98.20:38457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 18:37:56.429646 5230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:56.429925 5230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:56.429984 5230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:56.433974 5230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:56.434028 5230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:56.434094 5230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:37:56.438791 5230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:56.439821 5230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:56.441172 5230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:56.450502 5235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:56.450526 5238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:56.450526 5236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:56.450766 5230 server_base.cc:1034] running on GCE node
I20260619 18:37:56.451225 5230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:56.451843 5230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:56.452993 5230 hybrid_clock.cc:648] HybridClock initialized: now 1781894276452963 us; error 41 us; skew 500 ppm
I20260619 18:37:56.458277 5230 webserver.cc:460] Webserver started at http://127.0.98.2:34939/ using document root <none> and password file <none>
I20260619 18:37:56.458832 5230 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:56.458891 5230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:56.459069 5230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:56.461056 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "299f50970fe94ce987e1f6b7bd3711c1"
format_stamp: "Formatted at 2026-06-19 18:37:56 on dist-test-slave-rxv6"
server_key: "e2f1be7e64d21592c217c6a7dd4fcaa7"
server_key_iv: "204bf1a8fe5ff5b9fc29d32be0cadac2"
server_key_version: "encryptionkey@0"
I20260619 18:37:56.461583 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "299f50970fe94ce987e1f6b7bd3711c1"
format_stamp: "Formatted at 2026-06-19 18:37:56 on dist-test-slave-rxv6"
server_key: "e2f1be7e64d21592c217c6a7dd4fcaa7"
server_key_iv: "204bf1a8fe5ff5b9fc29d32be0cadac2"
server_key_version: "encryptionkey@0"
I20260619 18:37:56.465111 5230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:37:56.467375 5244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:56.468511 5230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:37:56.468598 5230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "299f50970fe94ce987e1f6b7bd3711c1"
format_stamp: "Formatted at 2026-06-19 18:37:56 on dist-test-slave-rxv6"
server_key: "e2f1be7e64d21592c217c6a7dd4fcaa7"
server_key_iv: "204bf1a8fe5ff5b9fc29d32be0cadac2"
server_key_version: "encryptionkey@0"
I20260619 18:37:56.468700 5230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:56.472909 5230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:56.473433 5230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:56.473575 5230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:56.474100 5230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:56.474903 5230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:56.474947 5230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:56.474982 5230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:56.474998 5230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:56.482846 5230 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39259
I20260619 18:37:56.482875 5357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39259 every 8 connection(s)
I20260619 18:37:56.483662 5230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:37:56.489059 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.485309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41187 (local address 127.0.98.62:38411)
0619 18:37:56.485432 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:56.485436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:56.486381 (+ 945us) server_negotiation.cc:407] Connection header received
0619 18:37:56.487619 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:56.487622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:56.487657 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:56.487745 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:56.488741 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:56.488744 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:56.488746 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:37:56.488755 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:56.488772 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:37:56.488779 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:56.488847 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:56.488931 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 18:37:56.488962 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:37:56.489853 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.485653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38411 (local address 127.0.98.2:41187)
0619 18:37:56.486265 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:56.486302 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:56.487437 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:56.487882 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:56.487889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:56.487971 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:56.488479 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:37:56.488480 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:56.488517 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:37:56.488616 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:56.488840 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:56.488844 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:56.488847 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 18:37:56.488962 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 18:37:56.489139 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":96,"threads_started":1}
I20260619 18:37:56.490304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 5230
I20260619 18:37:56.490420 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:37:56.490727 392 external_mini_cluster.cc:1428] Setting key c8db94544ef83fb8e83dec8df765e08d
I20260619 18:37:56.490948 5358 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38411
I20260619 18:37:56.491160 5358 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:56.491611 5358 heartbeater.cc:511] Master 127.0.98.62:38411 requested a full tablet report, sending...
I20260619 18:37:56.492558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38411
--builtin_ntp_servers=127.0.98.20:38457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 18:37:56.492551 5038 ts_manager.cc:194] Registered new tserver with Master: 299f50970fe94ce987e1f6b7bd3711c1 (127.0.98.2:39259)
W20260619 18:37:56.599213 5361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:56.599466 5361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:56.599520 5361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:56.603579 5361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:56.603649 5361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:56.603726 5361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:37:56.608513 5361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:56.609790 5361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:56.611328 5361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:56.620323 5369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:56.620323 5367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:56.620349 5366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:56.620410 5361 server_base.cc:1034] running on GCE node
I20260619 18:37:56.621021 5361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:56.621590 5361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:56.622746 5361 hybrid_clock.cc:648] HybridClock initialized: now 1781894276622729 us; error 28 us; skew 500 ppm
I20260619 18:37:56.624462 5361 webserver.cc:460] Webserver started at http://127.0.98.3:33089/ using document root <none> and password file <none>
I20260619 18:37:56.625001 5361 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:56.625061 5361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:56.625212 5361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:56.627189 5361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "d8f533a1b27447e88f1e5090475829c3"
format_stamp: "Formatted at 2026-06-19 18:37:56 on dist-test-slave-rxv6"
server_key: "864772400539851ff35a8a1691f26c80"
server_key_iv: "42c9491ba639bc6fda788cf8a4ce7eac"
server_key_version: "encryptionkey@0"
I20260619 18:37:56.627676 5361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d8f533a1b27447e88f1e5090475829c3"
format_stamp: "Formatted at 2026-06-19 18:37:56 on dist-test-slave-rxv6"
server_key: "864772400539851ff35a8a1691f26c80"
server_key_iv: "42c9491ba639bc6fda788cf8a4ce7eac"
server_key_version: "encryptionkey@0"
I20260619 18:37:56.631071 5361 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:37:56.633322 5375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:56.634483 5361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:37:56.634572 5361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "d8f533a1b27447e88f1e5090475829c3"
format_stamp: "Formatted at 2026-06-19 18:37:56 on dist-test-slave-rxv6"
server_key: "864772400539851ff35a8a1691f26c80"
server_key_iv: "42c9491ba639bc6fda788cf8a4ce7eac"
server_key_version: "encryptionkey@0"
I20260619 18:37:56.634654 5361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:56.638881 5361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:56.639412 5361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:56.639541 5361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:56.640023 5361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:56.640901 5361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:56.640945 5361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:56.640981 5361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:56.641000 5361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:56.649261 5361 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32839
I20260619 18:37:56.649286 5488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32839 every 8 connection(s)
I20260619 18:37:56.650154 5361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:37:56.655747 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.651840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42751 (local address 127.0.98.62:38411)
0619 18:37:56.651969 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:56.651973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:56.653018 (+ 1045us) server_negotiation.cc:407] Connection header received
0619 18:37:56.654289 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:56.654292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:56.654329 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:56.654408 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:56.655389 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:56.655392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:56.655394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:37:56.655405 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:56.655426 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:37:56.655434 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:56.655511 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:56.655609 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 18:37:56.655644 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:37:56.656566 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.652243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38411 (local address 127.0.98.3:42751)
0619 18:37:56.652869 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:56.652908 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:56.654094 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:56.654549 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:56.654556 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:56.654640 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:56.655142 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:37:56.655143 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:56.655182 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:37:56.655244 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:56.655505 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:56.655510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:56.655514 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 18:37:56.655633 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 18:37:56.655836 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":103,"threads_started":1}
I20260619 18:37:56.657645 5489 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38411
I20260619 18:37:56.657842 5489 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:56.658244 5489 heartbeater.cc:511] Master 127.0.98.62:38411 requested a full tablet report, sending...
I20260619 18:37:56.658277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 5361
I20260619 18:37:56.658373 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:37:56.658631 392 external_mini_cluster.cc:1428] Setting key ac6d586a2f13af35d970a03cbbd846aa
I20260619 18:37:56.659256 5038 ts_manager.cc:194] Registered new tserver with Master: d8f533a1b27447e88f1e5090475829c3 (127.0.98.3:32839)
I20260619 18:37:56.660547 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:37:56.663925 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.662334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47452 (local address 127.0.98.62:38411)
0619 18:37:56.662435 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:56.662440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:56.662697 (+ 257us) server_negotiation.cc:407] Connection header received
0619 18:37:56.662833 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:56.662837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:56.662871 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:56.662931 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:56.663574 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:56.663577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:56.663580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:37:56.663590 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:56.663609 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:37:56.663618 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:56.663685 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:56.663771 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 18:37:56.663810 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 18:37:56.666528 5038 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47452:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:37:56.669348 5038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:37:56.680138 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.677086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39259 (local address 127.0.0.1:44688)
0619 18:37:56.677934 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:56.677983 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:56.678085 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:56.678820 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:56.678826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:56.678898 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:56.679443 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:37:56.679445 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:56.679460 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:37:56.679522 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:56.679871 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:56.679879 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:56.679885 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:37:56.679970 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 18:37:56.680031 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"spinlock_wait_cycles":17792,"thread_start_us":80,"threads_started":1}
I20260619 18:37:56.680709 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.677136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32839 (local address 127.0.0.1:36628)
0619 18:37:56.677934 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:56.677983 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:56.678085 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:56.679405 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:56.679408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:56.679423 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:56.679839 (+ 416us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:37:56.679841 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:56.679852 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:37:56.680070 (+ 218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:56.680346 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:56.680348 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:56.680351 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 18:37:56.680402 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 18:37:56.680433 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"spinlock_wait_cycles":7168,"thread_start_us":107,"threads_started":1}
I20260619 18:37:56.680708 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.677600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43721 (local address 127.0.0.1:39632)
0619 18:37:56.678446 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:56.678455 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:56.678527 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:56.679497 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:56.679500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:56.679513 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:56.680213 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:37:56.680214 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:56.680224 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:37:56.680244 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:56.680529 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:56.680532 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:56.680535 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 18:37:56.680583 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 18:37:56.680614 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":90,"threads_started":1}
I20260619 18:37:56.681053 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.677205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.0.98.2:39259)
0619 18:37:56.678454 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:56.678463 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:56.678484 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:37:56.678531 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:56.678536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:56.678684 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:56.678808 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:56.679663 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:56.679669 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:56.679681 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:37:56.679692 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:56.679740 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:37:56.679774 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:56.680509 (+ 735us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:56.680560 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 18:37:56.680630 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1156,"thread_start_us":87,"threads_started":1}
I20260619 18:37:56.681226 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.678034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39632 (local address 127.0.98.1:43721)
0619 18:37:56.678347 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:56.678359 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:56.678529 (+ 170us) server_negotiation.cc:407] Connection header received
0619 18:37:56.678608 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:56.678614 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:56.678790 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:56.678911 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:56.680360 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:56.680364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:56.680374 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:37:56.680385 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:56.680420 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:37:56.680444 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:56.680945 (+ 501us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:56.680991 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 18:37:56.681058 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":116,"threads_started":1}
I20260619 18:37:56.681236 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.677452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36628 (local address 127.0.98.3:32839)
0619 18:37:56.678761 (+ 1309us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:56.678771 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:56.678791 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:37:56.678839 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:56.678843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:56.678980 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:56.679624 (+ 644us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:56.680165 (+ 541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:56.680171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:56.680182 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:37:56.680192 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:56.680229 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:37:56.680254 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:56.681010 (+ 756us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:56.681047 (+ 37us) server_negotiation.cc:299] Negotiation successful
0619 18:37:56.681095 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1207,"thread_start_us":90,"threads_started":1}
I20260619 18:37:56.683557 5161 tablet_service.cc:1467] Processing CreateTablet for tablet 3ed1a248c8514f2e9214da485d292fc8 (DEFAULT_TABLE table=test-table [id=3789046dca6d4774814e6e722b47de51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:56.683748 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 3ed1a248c8514f2e9214da485d292fc8 (DEFAULT_TABLE table=test-table [id=3789046dca6d4774814e6e722b47de51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:56.683748 5292 tablet_service.cc:1467] Processing CreateTablet for tablet 3ed1a248c8514f2e9214da485d292fc8 (DEFAULT_TABLE table=test-table [id=3789046dca6d4774814e6e722b47de51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:37:56.684592 5161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ed1a248c8514f2e9214da485d292fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:56.684708 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ed1a248c8514f2e9214da485d292fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:56.684708 5292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ed1a248c8514f2e9214da485d292fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:56.690425 5506 tablet_bootstrap.cc:492] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d: Bootstrap starting.
I20260619 18:37:56.691920 5507 tablet_bootstrap.cc:492] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1: Bootstrap starting.
I20260619 18:37:56.692126 5508 tablet_bootstrap.cc:492] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3: Bootstrap starting.
I20260619 18:37:56.692798 5506 tablet_bootstrap.cc:654] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:56.693629 5506 log.cc:824] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:56.693863 5507 tablet_bootstrap.cc:654] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:56.694129 5508 tablet_bootstrap.cc:654] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:56.694711 5507 log.cc:824] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:56.694818 5508 log.cc:824] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:56.695719 5506 tablet_bootstrap.cc:492] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d: No bootstrap required, opened a new log
I20260619 18:37:56.695899 5506 ts_tablet_manager.cc:1397] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:37:56.696530 5508 tablet_bootstrap.cc:492] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3: No bootstrap required, opened a new log
I20260619 18:37:56.696530 5507 tablet_bootstrap.cc:492] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1: No bootstrap required, opened a new log
I20260619 18:37:56.696702 5508 ts_tablet_manager.cc:1397] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:37:56.696712 5507 ts_tablet_manager.cc:1397] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:37:56.699510 5506 raft_consensus.cc:348] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299f50970fe94ce987e1f6b7bd3711c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39259 } } peers { permanent_uuid: "d8f533a1b27447e88f1e5090475829c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32839 } } peers { permanent_uuid: "c7986d7213914077bab6744a9e26b80d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43721 } }
I20260619 18:37:56.699745 5506 raft_consensus.cc:374] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:56.699796 5506 raft_consensus.cc:729] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7986d7213914077bab6744a9e26b80d, State: Initialized, Role: FOLLOWER
I20260619 18:37:56.700273 5506 consensus_queue.cc:260] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299f50970fe94ce987e1f6b7bd3711c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39259 } } peers { permanent_uuid: "d8f533a1b27447e88f1e5090475829c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32839 } } peers { permanent_uuid: "c7986d7213914077bab6744a9e26b80d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43721 } }
I20260619 18:37:56.700533 5512 raft_consensus.cc:482] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:37:56.700695 5512 raft_consensus.cc:504] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299f50970fe94ce987e1f6b7bd3711c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39259 } } peers { permanent_uuid: "d8f533a1b27447e88f1e5090475829c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32839 } } peers { permanent_uuid: "c7986d7213914077bab6744a9e26b80d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43721 } }
I20260619 18:37:56.700898 5508 raft_consensus.cc:348] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299f50970fe94ce987e1f6b7bd3711c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39259 } } peers { permanent_uuid: "d8f533a1b27447e88f1e5090475829c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32839 } } peers { permanent_uuid: "c7986d7213914077bab6744a9e26b80d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43721 } }
I20260619 18:37:56.700898 5507 raft_consensus.cc:348] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299f50970fe94ce987e1f6b7bd3711c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39259 } } peers { permanent_uuid: "d8f533a1b27447e88f1e5090475829c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32839 } } peers { permanent_uuid: "c7986d7213914077bab6744a9e26b80d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43721 } }
I20260619 18:37:56.701148 5507 raft_consensus.cc:374] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:56.701148 5508 raft_consensus.cc:374] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:56.701208 5507 raft_consensus.cc:729] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299f50970fe94ce987e1f6b7bd3711c1, State: Initialized, Role: FOLLOWER
I20260619 18:37:56.701216 5508 raft_consensus.cc:729] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f533a1b27447e88f1e5090475829c3, State: Initialized, Role: FOLLOWER
I20260619 18:37:56.701336 5506 ts_tablet_manager.cc:1428] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260619 18:37:56.701686 5227 heartbeater.cc:503] Master 127.0.98.62:38411 was elected leader, sending a full tablet report...
I20260619 18:37:56.701643 5507 consensus_queue.cc:260] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299f50970fe94ce987e1f6b7bd3711c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39259 } } peers { permanent_uuid: "d8f533a1b27447e88f1e5090475829c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32839 } } peers { permanent_uuid: "c7986d7213914077bab6744a9e26b80d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43721 } }
I20260619 18:37:56.701714 5508 consensus_queue.cc:260] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299f50970fe94ce987e1f6b7bd3711c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39259 } } peers { permanent_uuid: "d8f533a1b27447e88f1e5090475829c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32839 } } peers { permanent_uuid: "c7986d7213914077bab6744a9e26b80d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43721 } }
I20260619 18:37:56.702601 5507 ts_tablet_manager.cc:1428] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260619 18:37:56.702601 5508 ts_tablet_manager.cc:1428] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 18:37:56.702920 5489 heartbeater.cc:503] Master 127.0.98.62:38411 was elected leader, sending a full tablet report...
I20260619 18:37:56.703074 5358 heartbeater.cc:503] Master 127.0.98.62:38411 was elected leader, sending a full tablet report...
I20260619 18:37:56.703186 5512 leader_election.cc:288] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 299f50970fe94ce987e1f6b7bd3711c1 (127.0.98.2:39259), d8f533a1b27447e88f1e5090475829c3 (127.0.98.3:32839)
I20260619 18:37:56.707373 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.705102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41779 (local address 127.0.98.3:32839)
0619 18:37:56.705218 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:56.705222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:56.705815 (+ 593us) server_negotiation.cc:407] Connection header received
0619 18:37:56.705858 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:56.705860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:56.705897 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:56.706005 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:56.706874 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:56.706878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:56.706880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:37:56.706892 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:56.706940 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:37:56.706951 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:56.707046 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:56.707208 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 18:37:56.707250 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:37:56.707423 5229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.705038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32839 (local address 127.0.98.1:41779)
0619 18:37:56.705641 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:56.705657 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:56.705733 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:56.706103 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:56.706105 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:56.706121 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:56.706618 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:37:56.706620 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:56.706658 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:37:56.706687 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:56.707109 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:56.707112 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:56.707116 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 18:37:56.707228 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 18:37:56.707288 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528}
I20260619 18:37:56.708169 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed1a248c8514f2e9214da485d292fc8" candidate_uuid: "c7986d7213914077bab6744a9e26b80d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8f533a1b27447e88f1e5090475829c3" is_pre_election: true
I20260619 18:37:56.708352 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.705435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32891 (local address 127.0.98.2:39259)
0619 18:37:56.705535 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:56.705540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:56.706775 (+ 1235us) server_negotiation.cc:407] Connection header received
0619 18:37:56.706920 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:56.706923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:56.706956 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:56.707002 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:56.707922 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:56.707926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:56.707928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:37:56.707941 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:56.707960 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:37:56.707970 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:56.708045 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:56.708188 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 18:37:56.708237 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 18:37:56.708617 5443 raft_consensus.cc:2454] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7986d7213914077bab6744a9e26b80d in term 0.
I20260619 18:37:56.708709 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.705373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39259 (local address 127.0.98.1:32891)
0619 18:37:56.706695 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:56.706717 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:56.706825 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:56.707108 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:56.707111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:56.707125 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:56.707756 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:37:56.707758 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:56.707768 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:37:56.707792 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:56.708085 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:56.708088 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:56.708092 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 18:37:56.708519 (+ 427us) client_negotiation.cc:240] Negotiation successful
0619 18:37:56.708552 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1264,"thread_start_us":103,"threads_started":1}
I20260619 18:37:56.709182 5117 leader_election.cc:302] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7986d7213914077bab6744a9e26b80d, d8f533a1b27447e88f1e5090475829c3; no voters:
I20260619 18:37:56.709225 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed1a248c8514f2e9214da485d292fc8" candidate_uuid: "c7986d7213914077bab6744a9e26b80d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299f50970fe94ce987e1f6b7bd3711c1" is_pre_election: true
I20260619 18:37:56.709455 5512 raft_consensus.cc:2781] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:37:56.709467 5312 raft_consensus.cc:2454] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7986d7213914077bab6744a9e26b80d in term 0.
I20260619 18:37:56.709550 5512 raft_consensus.cc:482] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:37:56.709600 5512 raft_consensus.cc:3037] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:56.710832 5512 raft_consensus.cc:504] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299f50970fe94ce987e1f6b7bd3711c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39259 } } peers { permanent_uuid: "d8f533a1b27447e88f1e5090475829c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32839 } } peers { permanent_uuid: "c7986d7213914077bab6744a9e26b80d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43721 } }
I20260619 18:37:56.711133 5512 leader_election.cc:288] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [CANDIDATE]: Term 1 election: Requested vote from peers 299f50970fe94ce987e1f6b7bd3711c1 (127.0.98.2:39259), d8f533a1b27447e88f1e5090475829c3 (127.0.98.3:32839)
I20260619 18:37:56.711372 5312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed1a248c8514f2e9214da485d292fc8" candidate_uuid: "c7986d7213914077bab6744a9e26b80d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299f50970fe94ce987e1f6b7bd3711c1"
I20260619 18:37:56.711444 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ed1a248c8514f2e9214da485d292fc8" candidate_uuid: "c7986d7213914077bab6744a9e26b80d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8f533a1b27447e88f1e5090475829c3"
I20260619 18:37:56.711484 5312 raft_consensus.cc:3037] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:56.711547 5443 raft_consensus.cc:3037] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:56.712662 5312 raft_consensus.cc:2454] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7986d7213914077bab6744a9e26b80d in term 1.
I20260619 18:37:56.712662 5443 raft_consensus.cc:2454] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7986d7213914077bab6744a9e26b80d in term 1.
I20260619 18:37:56.712925 5114 leader_election.cc:302] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 299f50970fe94ce987e1f6b7bd3711c1, c7986d7213914077bab6744a9e26b80d, d8f533a1b27447e88f1e5090475829c3; no voters:
I20260619 18:37:56.713136 5512 raft_consensus.cc:2781] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:56.713227 5512 raft_consensus.cc:686] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [term 1 LEADER]: Becoming Leader. State: Replica: c7986d7213914077bab6744a9e26b80d, State: Running, Role: LEADER
I20260619 18:37:56.713547 5512 consensus_queue.cc:237] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299f50970fe94ce987e1f6b7bd3711c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39259 } } peers { permanent_uuid: "d8f533a1b27447e88f1e5090475829c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32839 } } peers { permanent_uuid: "c7986d7213914077bab6744a9e26b80d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43721 } }
I20260619 18:37:56.716696 5039 catalog_manager.cc:5495] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d reported cstate change: term changed from 0 to 1, leader changed from <none> to c7986d7213914077bab6744a9e26b80d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c7986d7213914077bab6744a9e26b80d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299f50970fe94ce987e1f6b7bd3711c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8f533a1b27447e88f1e5090475829c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7986d7213914077bab6744a9e26b80d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43721 } health_report { overall_health: HEALTHY } } }
I20260619 18:37:56.730139 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.728794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39636 (local address 127.0.98.1:43721)
0619 18:37:56.728912 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:56.728915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:56.728930 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:37:56.729006 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:56.729009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:56.729091 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:56.729158 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:56.729760 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:56.729763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:56.729765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:37:56.729775 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:56.729795 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:37:56.729803 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:56.729877 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:56.729949 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 18:37:56.729987 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 18:37:56.734503 5359 tablet.cc:2369] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:37:56.736069 5443 raft_consensus.cc:1261] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3 [term 1 FOLLOWER]: Refusing update from remote peer c7986d7213914077bab6744a9e26b80d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:56.736068 5312 raft_consensus.cc:1261] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1 [term 1 FOLLOWER]: Refusing update from remote peer c7986d7213914077bab6744a9e26b80d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:37:56.736734 5513 consensus_queue.cc:1035] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8f533a1b27447e88f1e5090475829c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:56.736910 5512 consensus_queue.cc:1035] T 3ed1a248c8514f2e9214da485d292fc8 P c7986d7213914077bab6744a9e26b80d [LEADER]: Connected to new peer: Peer: permanent_uuid: "299f50970fe94ce987e1f6b7bd3711c1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:37:56.745812 5520 mvcc.cc:205] Tried to move back new op lower bound from 7298638957503696896 to 7298638957422006272. Current Snapshot: MvccSnapshot[applied={T|T < 7298638957503696896}]
I20260619 18:37:56.752295 5039 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:37:56.752434 5039 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47452:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:37:56.754587 5039 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e00f3210bf4842b9b8d2e259c493a933: Sending DeleteTablet for 3 replicas of tablet 3ed1a248c8514f2e9214da485d292fc8
I20260619 18:37:56.755308 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ed1a248c8514f2e9214da485d292fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:56 UTC) from {username='slave'} at 127.0.0.1:44688
I20260619 18:37:56.755307 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ed1a248c8514f2e9214da485d292fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:37:56 UTC) from {username='slave'} at 127.0.0.1:36628
I20260619 18:37:56.755652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 5099
I20260619 18:37:56.756937 5530 tablet_replica.cc:330] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1: stopping tablet replica
I20260619 18:37:56.757244 5530 raft_consensus.cc:2229] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:37:56.757511 5530 raft_consensus.cc:2258] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:56.757856 5529 tablet_replica.cc:330] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3: stopping tablet replica
I20260619 18:37:56.758113 5529 raft_consensus.cc:2229] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:37:56.758351 5529 raft_consensus.cc:2258] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:37:56.759757 5530 ts_tablet_manager.cc:1905] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:56.759779 5529 ts_tablet_manager.cc:1905] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:37:56.761843 5530 ts_tablet_manager.cc:1918] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:37:56.761960 5530 log.cc:1196] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/3ed1a248c8514f2e9214da485d292fc8
I20260619 18:37:56.762061 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 5230
I20260619 18:37:56.762234 5530 ts_tablet_manager.cc:1939] T 3ed1a248c8514f2e9214da485d292fc8 P 299f50970fe94ce987e1f6b7bd3711c1: Deleting consensus metadata
W20260619 18:37:56.762804 5024 connection.cc:535] client connection to 127.0.98.1:43721 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 18:37:56.762940 5024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 18:37:56.763433 5529 ts_tablet_manager.cc:1918] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:37:56.763542 5529 log.cc:1196] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/3ed1a248c8514f2e9214da485d292fc8
I20260619 18:37:56.763828 5529 ts_tablet_manager.cc:1939] T 3ed1a248c8514f2e9214da485d292fc8 P d8f533a1b27447e88f1e5090475829c3: Deleting consensus metadata
I20260619 18:37:56.763833 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.763555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43721 (local address 127.0.0.1:39648)
0619 18:37:56.763712 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:56.763767 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260619 18:37:56.763983 5024 catalog_manager.cc:4584] TS c7986d7213914077bab6744a9e26b80d (127.0.98.1:43721): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3ed1a248c8514f2e9214da485d292fc8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43721: connect: Connection refused (error 111)
I20260619 18:37:56.765014 5026 catalog_manager.cc:4841] TS d8f533a1b27447e88f1e5090475829c3 (127.0.98.3:32839): tablet 3ed1a248c8514f2e9214da485d292fc8 (table test-table [id=3789046dca6d4774814e6e722b47de51]) successfully deleted
I20260619 18:37:56.768159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 5361
I20260619 18:37:56.769243 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.768325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39259 (local address 127.0.0.1:44698)
0619 18:37:56.769112 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:56.769165 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":713}
I20260619 18:37:56.772758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 5008
2026-06-19T18:37:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (881 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 18:37:56.792284 392 mini_oidc.cc:152] Starting JWKS server
I20260619 18:37:56.794427 392 webserver.cc:460] Webserver started at https://127.0.0.1:44449/ using document root <none> and password file <none>
I20260619 18:37:56.794586 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 18:37:56.794833 392 webserver.cc:460] Webserver started at http://127.0.0.1:39865/ using document root <none> and password file <none>
2026-06-19T18:37:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:37:56Z Disabled control of system clock
I20260619 18:37:56.804544 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45897
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45897
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44449/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/testchainca.cert with env {}
W20260619 18:37:56.912750 5537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:56.913043 5537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:56.913082 5537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:56.917106 5537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:37:56.917171 5537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:56.917184 5537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:56.917200 5537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:37:56.917213 5537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:37:56.917228 5537 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:37:56.917240 5537 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44449/jwks.json
I20260619 18:37:56.922701 5537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44449/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:56.923751 5537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:56.924724 5537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:56.933156 5542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:56.933151 5543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:56.933151 5545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:56.933516 5537 server_base.cc:1034] running on GCE node
I20260619 18:37:56.934085 5537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:56.935216 5537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:56.936376 5537 hybrid_clock.cc:648] HybridClock initialized: now 1781894276936354 us; error 35 us; skew 500 ppm
I20260619 18:37:56.938253 5537 webserver.cc:460] Webserver started at http://127.0.98.62:36255/ using document root <none> and password file <none>
I20260619 18:37:56.938812 5537 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:56.938879 5537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:56.939066 5537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:56.941031 5537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "ca7f580c049045fb9b34c1ab91f5d135"
format_stamp: "Formatted at 2026-06-19 18:37:56 on dist-test-slave-rxv6"
server_key: "082a976ebe50cd7c2a19fcb762b5cf80"
server_key_iv: "89eb8eaeb8fff49141aa4c765397ff38"
server_key_version: "encryptionkey@0"
I20260619 18:37:56.941551 5537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca7f580c049045fb9b34c1ab91f5d135"
format_stamp: "Formatted at 2026-06-19 18:37:56 on dist-test-slave-rxv6"
server_key: "082a976ebe50cd7c2a19fcb762b5cf80"
server_key_iv: "89eb8eaeb8fff49141aa4c765397ff38"
server_key_version: "encryptionkey@0"
I20260619 18:37:56.945106 5537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:37:56.947453 5551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:56.948733 5537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:37:56.948861 5537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "ca7f580c049045fb9b34c1ab91f5d135"
format_stamp: "Formatted at 2026-06-19 18:37:56 on dist-test-slave-rxv6"
server_key: "082a976ebe50cd7c2a19fcb762b5cf80"
server_key_iv: "89eb8eaeb8fff49141aa4c765397ff38"
server_key_version: "encryptionkey@0"
I20260619 18:37:56.948967 5537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:56.963861 5537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:56.964570 5537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:56.964743 5537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:56.971010 5537 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45897
I20260619 18:37:56.971017 5603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45897 every 8 connection(s)
I20260619 18:37:56.971907 5537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:37:56.975111 5604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:56.980312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 5537
I20260619 18:37:56.980397 5604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135: Bootstrap starting.
I20260619 18:37:56.980430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:37:56.980677 392 external_mini_cluster.cc:1428] Setting key 2200bd44947ae7560033d69d489fe5aa
I20260619 18:37:56.982874 5604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:56.983577 5604 log.cc:824] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:56.985251 5604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135: No bootstrap required, opened a new log
I20260619 18:37:56.988521 5604 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f580c049045fb9b34c1ab91f5d135" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } }
I20260619 18:37:56.988781 5604 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:56.988835 5604 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7f580c049045fb9b34c1ab91f5d135, State: Initialized, Role: FOLLOWER
I20260619 18:37:56.989252 5604 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f580c049045fb9b34c1ab91f5d135" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } }
I20260619 18:37:56.989367 5604 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:37:56.989421 5604 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:37:56.989511 5604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:56.989516 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:56.981967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.0.98.62:45897)
0619 18:37:56.982590 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:56.982604 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:56.982639 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:37:56.983320 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:56.983340 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:56.983775 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:56.984076 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:56.984495 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:56.985443 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:56.986380 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:56.986850 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:56.986993 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:56.987760 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:56.987766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:56.987777 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:37:56.988146 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:56.988202 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:37:56.988221 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:56.988305 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:56.988441 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 18:37:56.988719 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":136,"threads_started":1}
I20260619 18:37:56.990382 5604 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f580c049045fb9b34c1ab91f5d135" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } }
I20260619 18:37:56.990711 5604 leader_election.cc:302] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca7f580c049045fb9b34c1ab91f5d135; no voters:
I20260619 18:37:56.991009 5604 leader_election.cc:288] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:37:56.991108 5609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:56.991281 5609 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [term 1 LEADER]: Becoming Leader. State: Replica: ca7f580c049045fb9b34c1ab91f5d135, State: Running, Role: LEADER
I20260619 18:37:56.991582 5609 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f580c049045fb9b34c1ab91f5d135" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } }
I20260619 18:37:56.992180 5604 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:37:56.992486 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:37:56.992714 5611 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca7f580c049045fb9b34c1ab91f5d135. Latest consensus state: current_term: 1 leader_uuid: "ca7f580c049045fb9b34c1ab91f5d135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f580c049045fb9b34c1ab91f5d135" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } } }
I20260619 18:37:56.992817 5611 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:56.993556 5610 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca7f580c049045fb9b34c1ab91f5d135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca7f580c049045fb9b34c1ab91f5d135" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45897 } } }
I20260619 18:37:56.993651 5610 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135 [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:56.994633 5614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:37:56.997102 5614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:37:57.002354 5614 catalog_manager.cc:1347] Generated new cluster ID: 77a6fc73390b41918f2a1736836d4822
I20260619 18:37:57.002419 5614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:37:57.025269 5614 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:37:57.026275 5614 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:37:57.040177 5614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca7f580c049045fb9b34c1ab91f5d135: Generated new TSK 0
I20260619 18:37:57.040872 5614 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 18:37:57.075191 5634 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 18:37:57.075313 5634 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 18:37:57.075747 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:57.051447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38702 (local address 127.0.98.62:45897)
0619 18:37:57.051573 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:57.051577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:57.051788 (+ 211us) server_negotiation.cc:407] Connection header received
0619 18:37:57.052003 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:57.052006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:57.052051 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:57.052137 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:37:57.052454 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.053082 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.054370 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.054723 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.054824 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:57.055176 (+ 352us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:37:57.075319 (+ 20143us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 18:37:57.075624 (+ 305us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38702: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 18:37:57.075824 5607 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38702: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 18:37:57.077148 5633 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:37:57.051372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45897 (local address 127.0.0.1:38702)
0619 18:37:57.051703 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:57.051719 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:57.051854 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:57.052171 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:57.052174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:57.052184 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:37:57.052321 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:57.052327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.053217 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.053221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:57.054251 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:57.054261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.054826 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.054847 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:57.054950 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:57.055004 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:37:57.075588 (+ 20584us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 18:37:57.075834 (+ 246us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45897: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":89,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:45897: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44449/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 18:37:57.077930 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 5537
2026-06-19T18:37:57Z chronyd exiting
I20260619 18:37:57.114027 392 test_util.cc:175] -----------------------------------------------
I20260619 18:37:57.114110 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781894249862417-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (323 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 18:37:57.115082 392 mini_oidc.cc:152] Starting JWKS server
I20260619 18:37:57.115936 392 webserver.cc:460] Webserver started at https://127.0.0.1:45171/ using document root <none> and password file <none>
I20260619 18:37:57.116032 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 18:37:57.116271 392 webserver.cc:460] Webserver started at http://127.0.0.1:41847/ using document root <none> and password file <none>
2026-06-19T18:37:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:37:57Z Disabled control of system clock
I20260619 18:37:57.123901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43845
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43845
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45171/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/testchainca.cert with env {}
W20260619 18:37:57.228219 5640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:57.228469 5640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:57.228524 5640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:57.232483 5640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:37:57.232542 5640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:57.232563 5640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:57.232584 5640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:37:57.232609 5640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:37:57.232637 5640 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:37:57.232666 5640 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45171/jwks.json
I20260619 18:37:57.237838 5640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45171/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:57.239025 5640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:57.240099 5640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:57.247998 5646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:57.248031 5645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:57.248018 5648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:57.248293 5640 server_base.cc:1034] running on GCE node
I20260619 18:37:57.248802 5640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:57.249841 5640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:57.250998 5640 hybrid_clock.cc:648] HybridClock initialized: now 1781894277250974 us; error 35 us; skew 500 ppm
I20260619 18:37:57.252664 5640 webserver.cc:460] Webserver started at http://127.0.98.62:38789/ using document root <none> and password file <none>
I20260619 18:37:57.253217 5640 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:57.253278 5640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:57.253450 5640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:57.255529 5640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "a367274d621645d0bd72bf7f904c5c23"
format_stamp: "Formatted at 2026-06-19 18:37:57 on dist-test-slave-rxv6"
server_key: "889d920cd95d3b687795833507559a84"
server_key_iv: "c4402bb9b86128d87e685bcb93b75bd3"
server_key_version: "encryptionkey@0"
I20260619 18:37:57.256043 5640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "a367274d621645d0bd72bf7f904c5c23"
format_stamp: "Formatted at 2026-06-19 18:37:57 on dist-test-slave-rxv6"
server_key: "889d920cd95d3b687795833507559a84"
server_key_iv: "c4402bb9b86128d87e685bcb93b75bd3"
server_key_version: "encryptionkey@0"
I20260619 18:37:57.259486 5640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 18:37:57.261719 5654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:57.262918 5640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:37:57.263023 5640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "a367274d621645d0bd72bf7f904c5c23"
format_stamp: "Formatted at 2026-06-19 18:37:57 on dist-test-slave-rxv6"
server_key: "889d920cd95d3b687795833507559a84"
server_key_iv: "c4402bb9b86128d87e685bcb93b75bd3"
server_key_version: "encryptionkey@0"
I20260619 18:37:57.263109 5640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:57.273238 5640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:57.273831 5640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:57.274022 5640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:57.280225 5640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43845
I20260619 18:37:57.280239 5706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43845 every 8 connection(s)
I20260619 18:37:57.281116 5640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:37:57.284003 5707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:57.288676 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 5640
I20260619 18:37:57.288789 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:37:57.289072 392 external_mini_cluster.cc:1428] Setting key a2b7b826f37711425dbfa91f2d7fb0ae
I20260619 18:37:57.289459 5707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23: Bootstrap starting.
I20260619 18:37:57.291718 5707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:57.292485 5707 log.cc:824] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:57.294490 5707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23: No bootstrap required, opened a new log
I20260619 18:37:57.297772 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:57.290326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.0.98.62:43845)
0619 18:37:57.290930 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:57.290944 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:57.290978 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:37:57.291571 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:57.291586 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:57.291999 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:57.292276 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:57.292630 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.293699 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.294837 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.295278 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.295391 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:57.296171 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:57.296176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:57.296189 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:37:57.296545 (+ 356us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:57.296600 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:37:57.296618 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:57.296688 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:57.296836 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 18:37:57.297081 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
I20260619 18:37:57.298209 5707 raft_consensus.cc:348] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a367274d621645d0bd72bf7f904c5c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } }
I20260619 18:37:57.298409 5707 raft_consensus.cc:374] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:57.298456 5707 raft_consensus.cc:729] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a367274d621645d0bd72bf7f904c5c23, State: Initialized, Role: FOLLOWER
I20260619 18:37:57.298884 5707 consensus_queue.cc:260] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a367274d621645d0bd72bf7f904c5c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } }
I20260619 18:37:57.299029 5707 raft_consensus.cc:388] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:37:57.299088 5707 raft_consensus.cc:482] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:37:57.299182 5707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:57.300123 5707 raft_consensus.cc:504] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a367274d621645d0bd72bf7f904c5c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } }
I20260619 18:37:57.300501 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:37:57.300462 5707 leader_election.cc:302] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a367274d621645d0bd72bf7f904c5c23; no voters:
I20260619 18:37:57.300797 5707 leader_election.cc:288] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:37:57.300866 5712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:57.301054 5712 raft_consensus.cc:686] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [term 1 LEADER]: Becoming Leader. State: Replica: a367274d621645d0bd72bf7f904c5c23, State: Running, Role: LEADER
I20260619 18:37:57.301368 5712 consensus_queue.cc:237] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a367274d621645d0bd72bf7f904c5c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } }
I20260619 18:37:57.302109 5707 sys_catalog.cc:564] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:37:57.302675 5712 sys_catalog.cc:455] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a367274d621645d0bd72bf7f904c5c23. Latest consensus state: current_term: 1 leader_uuid: "a367274d621645d0bd72bf7f904c5c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a367274d621645d0bd72bf7f904c5c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } } }
I20260619 18:37:57.302794 5712 sys_catalog.cc:458] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:57.303042 5713 sys_catalog.cc:455] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a367274d621645d0bd72bf7f904c5c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a367274d621645d0bd72bf7f904c5c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } } }
I20260619 18:37:57.303139 5717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:37:57.303169 5713 sys_catalog.cc:458] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23 [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:57.306188 5717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:37:57.311221 5717 catalog_manager.cc:1347] Generated new cluster ID: dd92ad0aca534f96996ac1a9702300a1
I20260619 18:37:57.311291 5717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:37:57.319938 5717 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:37:57.321058 5717 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:37:57.325949 5717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a367274d621645d0bd72bf7f904c5c23: Generated new TSK 0
I20260619 18:37:57.326659 5717 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 18:37:57.359607 5737 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 18:37:57.359697 5737 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 18:37:57.360327 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:57.336691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.0.98.62:43845)
0619 18:37:57.336817 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:57.336822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:57.336991 (+ 169us) server_negotiation.cc:407] Connection header received
0619 18:37:57.337180 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:57.337183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:57.337229 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:57.337318 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:37:57.337596 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.338460 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.339698 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.339918 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.339996 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:57.340274 (+ 278us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:37:57.359956 (+ 19682us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 18:37:57.360212 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41090: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 18:37:57.360391 5710 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41090: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 18:37:57.360620 5736 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:37:57.336601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43845 (local address 127.0.0.1:41090)
0619 18:37:57.336908 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:57.336923 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:57.337032 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:57.337330 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:57.337333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:57.337340 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:37:57.337455 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:57.337461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.338583 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.338587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:57.339579 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:57.339588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.340002 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.340020 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:57.340109 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:57.340138 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:37:57.360225 (+ 20087us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 18:37:57.360447 (+ 222us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43845: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":97,"threads_started":1}
I20260619 18:37:57.361228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 5640
2026-06-19T18:37:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (284 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 18:37:57.399387 392 mini_oidc.cc:152] Starting JWKS server
I20260619 18:37:57.400213 392 webserver.cc:460] Webserver started at https://127.0.0.1:39109/ using document root <none> and password file <none>
I20260619 18:37:57.400295 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 18:37:57.400511 392 webserver.cc:460] Webserver started at http://127.0.0.1:34279/ using document root <none> and password file <none>
2026-06-19T18:37:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:37:57Z Disabled control of system clock
I20260619 18:37:57.409368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42185
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42185
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39109/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 18:37:57.513829 5743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:57.514132 5743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:57.514187 5743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:57.518270 5743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:37:57.518337 5743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:57.518360 5743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:57.518380 5743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:37:57.518402 5743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:37:57.518431 5743 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:37:57.518455 5743 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39109/jwks.json
I20260619 18:37:57.523243 5743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39109/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:57.524331 5743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:57.525671 5743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:57.533607 5749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:57.533743 5751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:57.533618 5748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:57.533836 5743 server_base.cc:1034] running on GCE node
I20260619 18:37:57.534507 5743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:57.535573 5743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:57.536760 5743 hybrid_clock.cc:648] HybridClock initialized: now 1781894277536744 us; error 32 us; skew 500 ppm
I20260619 18:37:57.538535 5743 webserver.cc:460] Webserver started at http://127.0.98.62:33863/ using document root <none> and password file <none>
I20260619 18:37:57.539060 5743 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:57.539124 5743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:57.539317 5743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:57.541244 5743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "ba3739ee63604945b1b34625b1d5ab9b"
format_stamp: "Formatted at 2026-06-19 18:37:57 on dist-test-slave-rxv6"
server_key: "83b1f43ca98d5bc6ad312d433ee22f54"
server_key_iv: "3dd07c1189bd0ec96b1ae0f61c2e3340"
server_key_version: "encryptionkey@0"
I20260619 18:37:57.541772 5743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba3739ee63604945b1b34625b1d5ab9b"
format_stamp: "Formatted at 2026-06-19 18:37:57 on dist-test-slave-rxv6"
server_key: "83b1f43ca98d5bc6ad312d433ee22f54"
server_key_iv: "3dd07c1189bd0ec96b1ae0f61c2e3340"
server_key_version: "encryptionkey@0"
I20260619 18:37:57.545204 5743 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:37:57.547673 5757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:57.548902 5743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:37:57.549015 5743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "ba3739ee63604945b1b34625b1d5ab9b"
format_stamp: "Formatted at 2026-06-19 18:37:57 on dist-test-slave-rxv6"
server_key: "83b1f43ca98d5bc6ad312d433ee22f54"
server_key_iv: "3dd07c1189bd0ec96b1ae0f61c2e3340"
server_key_version: "encryptionkey@0"
I20260619 18:37:57.549119 5743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:57.580173 5743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:57.580897 5743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:57.581072 5743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:57.587705 5743 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42185
I20260619 18:37:57.587718 5809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42185 every 8 connection(s)
I20260619 18:37:57.588796 5743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:37:57.591918 5810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:57.594587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 5743
I20260619 18:37:57.594734 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:37:57.595013 392 external_mini_cluster.cc:1428] Setting key a99bde1683a771ec871b076914c8057e
I20260619 18:37:57.597884 5810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b: Bootstrap starting.
I20260619 18:37:57.600528 5810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:57.601296 5810 log.cc:824] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:57.603252 5810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b: No bootstrap required, opened a new log
I20260619 18:37:57.605352 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:57.596553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58210 (local address 127.0.98.62:42185)
0619 18:37:57.597234 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:57.597254 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:57.597295 (+ 41us) server_negotiation.cc:407] Connection header received
0619 18:37:57.598111 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:57.598137 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:57.598622 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:57.598974 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:57.599415 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.600868 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.601751 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.602286 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.602470 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:57.603323 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:57.603330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:57.603353 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:37:57.603786 (+ 433us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:57.603853 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:37:57.603878 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:57.603963 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:57.604385 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 18:37:57.604647 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":112,"threads_started":1}
I20260619 18:37:57.606834 5810 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3739ee63604945b1b34625b1d5ab9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } }
I20260619 18:37:57.607105 5810 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:57.607169 5810 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba3739ee63604945b1b34625b1d5ab9b, State: Initialized, Role: FOLLOWER
I20260619 18:37:57.607585 5810 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3739ee63604945b1b34625b1d5ab9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } }
I20260619 18:37:57.607736 5810 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:37:57.607800 5810 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:37:57.607893 5810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:57.608282 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:37:57.608846 5810 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3739ee63604945b1b34625b1d5ab9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } }
I20260619 18:37:57.609174 5810 leader_election.cc:302] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba3739ee63604945b1b34625b1d5ab9b; no voters:
I20260619 18:37:57.609481 5810 leader_election.cc:288] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:37:57.609567 5815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:57.609732 5815 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [term 1 LEADER]: Becoming Leader. State: Replica: ba3739ee63604945b1b34625b1d5ab9b, State: Running, Role: LEADER
I20260619 18:37:57.610044 5815 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3739ee63604945b1b34625b1d5ab9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } }
I20260619 18:37:57.610725 5810 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:37:57.611150 5815 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba3739ee63604945b1b34625b1d5ab9b. Latest consensus state: current_term: 1 leader_uuid: "ba3739ee63604945b1b34625b1d5ab9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3739ee63604945b1b34625b1d5ab9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } }
I20260619 18:37:57.611256 5815 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:57.611536 5816 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba3739ee63604945b1b34625b1d5ab9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba3739ee63604945b1b34625b1d5ab9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } }
I20260619 18:37:57.611604 5819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:37:57.611636 5816 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:57.614648 5819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:37:57.620062 5819 catalog_manager.cc:1347] Generated new cluster ID: 45098fde03204684bc4ef428a535058a
I20260619 18:37:57.620126 5819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:37:57.633987 5819 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:37:57.634954 5819 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:37:57.640759 5819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba3739ee63604945b1b34625b1d5ab9b: Generated new TSK 0
I20260619 18:37:57.641424 5819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:37:57.661329 5813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:57.656237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58226 (local address 127.0.98.62:42185)
0619 18:37:57.656364 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:57.656369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:57.656561 (+ 192us) server_negotiation.cc:407] Connection header received
0619 18:37:57.656721 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:57.656724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:57.656773 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:57.656861 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:37:57.657090 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.657619 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.658916 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.659140 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.659232 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:57.659549 (+ 317us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:37:57.660944 (+ 1395us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39109/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 18:37:57.661183 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58226: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39109/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 18:37:57.661397 5813 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58226: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39109/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 18:37:57.663232 5839 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:37:57.656142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42185 (local address 127.0.0.1:58226)
0619 18:37:57.656447 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:57.656462 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:57.656582 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:57.656880 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:57.656883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:57.656890 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:37:57.657004 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:57.657011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.657712 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.657716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:57.658773 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:57.658784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.659223 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.659246 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:57.659333 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:57.659362 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:37:57.661103 (+ 1741us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39109/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 18:37:57.663060 (+ 1957us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42185: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39109/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":1683,"thread_start_us":103,"threads_started":1}
I20260619 18:37:57.663836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 5743
E20260619 18:37:57.672449 5840 webserver.cc:533] Webserver: sslize failed:
E20260619 18:37:57.672530 5840 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T18:37:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (302 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T18:37:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:37:57Z Disabled control of system clock
I20260619 18:37:57.709762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40643
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40643
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/testchainca.cert with env {}
W20260619 18:37:57.818719 5844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:57.818982 5844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:57.819021 5844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:57.822886 5844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:37:57.822935 5844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:57.822949 5844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:57.822963 5844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:37:57.822983 5844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 18:37:57.822997 5844 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 18:37:57.823009 5844 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 18:37:57.828231 5844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40643
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:57.829209 5844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:57.830188 5844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:57.838124 5850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:57.838158 5844 server_base.cc:1034] running on GCE node
W20260619 18:37:57.838174 5849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:57.838119 5852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:57.838883 5844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:57.839910 5844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:57.841074 5844 hybrid_clock.cc:648] HybridClock initialized: now 1781894277841051 us; error 37 us; skew 500 ppm
I20260619 18:37:57.842907 5844 webserver.cc:460] Webserver started at http://127.0.98.62:45917/ using document root <none> and password file <none>
I20260619 18:37:57.843420 5844 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:57.843475 5844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:57.843632 5844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:57.845548 5844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "729f1e21f7fb4f05a5d32a2cdf08ee8a"
format_stamp: "Formatted at 2026-06-19 18:37:57 on dist-test-slave-rxv6"
server_key: "236a0dc35d552b1888235f4be32d0707"
server_key_iv: "9f0aa36be76bcc1aae81ff2e0e160fc5"
server_key_version: "encryptionkey@0"
I20260619 18:37:57.846074 5844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "729f1e21f7fb4f05a5d32a2cdf08ee8a"
format_stamp: "Formatted at 2026-06-19 18:37:57 on dist-test-slave-rxv6"
server_key: "236a0dc35d552b1888235f4be32d0707"
server_key_iv: "9f0aa36be76bcc1aae81ff2e0e160fc5"
server_key_version: "encryptionkey@0"
I20260619 18:37:57.849464 5844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:37:57.851794 5858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:57.852969 5844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:37:57.853066 5844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "729f1e21f7fb4f05a5d32a2cdf08ee8a"
format_stamp: "Formatted at 2026-06-19 18:37:57 on dist-test-slave-rxv6"
server_key: "236a0dc35d552b1888235f4be32d0707"
server_key_iv: "9f0aa36be76bcc1aae81ff2e0e160fc5"
server_key_version: "encryptionkey@0"
I20260619 18:37:57.853168 5844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:57.874712 5844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:57.875392 5844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:57.875557 5844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:57.881975 5844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40643
I20260619 18:37:57.882015 5910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40643 every 8 connection(s)
I20260619 18:37:57.882859 5844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:37:57.885023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 5844
I20260619 18:37:57.885115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:37:57.885375 392 external_mini_cluster.cc:1428] Setting key 094027e9777f0132a2097561c9072d2d
I20260619 18:37:57.885695 5911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:57.892540 5911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a: Bootstrap starting.
I20260619 18:37:57.895131 5911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:57.895406 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:57.886698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34700 (local address 127.0.98.62:40643)
0619 18:37:57.887367 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:57.887385 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:57.887428 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:37:57.888284 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:57.888307 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:57.888798 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:57.889174 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:57.889604 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.891016 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.891897 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.892551 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.892698 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:57.893478 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:57.893484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:57.893498 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:37:57.894009 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:57.894072 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:37:57.894096 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:57.894183 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:57.894273 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 18:37:57.894515 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":129,"threads_started":1}
I20260619 18:37:57.895908 5911 log.cc:824] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:57.897675 5911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a: No bootstrap required, opened a new log
I20260619 18:37:57.897871 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:37:57.900954 5911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729f1e21f7fb4f05a5d32a2cdf08ee8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40643 } }
I20260619 18:37:57.901163 5911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:57.901219 5911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729f1e21f7fb4f05a5d32a2cdf08ee8a, State: Initialized, Role: FOLLOWER
I20260619 18:37:57.901604 5911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729f1e21f7fb4f05a5d32a2cdf08ee8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40643 } }
I20260619 18:37:57.901724 5911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:37:57.901805 5911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:37:57.901923 5911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:57.902756 5911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729f1e21f7fb4f05a5d32a2cdf08ee8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40643 } }
I20260619 18:37:57.903061 5911 leader_election.cc:302] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 729f1e21f7fb4f05a5d32a2cdf08ee8a; no voters:
I20260619 18:37:57.903326 5911 leader_election.cc:288] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:37:57.903450 5916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:57.903669 5916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [term 1 LEADER]: Becoming Leader. State: Replica: 729f1e21f7fb4f05a5d32a2cdf08ee8a, State: Running, Role: LEADER
I20260619 18:37:57.903976 5916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729f1e21f7fb4f05a5d32a2cdf08ee8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40643 } }
I20260619 18:37:57.904561 5911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:37:57.905117 5918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 729f1e21f7fb4f05a5d32a2cdf08ee8a. Latest consensus state: current_term: 1 leader_uuid: "729f1e21f7fb4f05a5d32a2cdf08ee8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729f1e21f7fb4f05a5d32a2cdf08ee8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40643 } } }
I20260619 18:37:57.905292 5918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:57.905558 5917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "729f1e21f7fb4f05a5d32a2cdf08ee8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "729f1e21f7fb4f05a5d32a2cdf08ee8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40643 } } }
I20260619 18:37:57.905665 5917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:57.905764 5925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:37:57.908809 5925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:37:57.914537 5925 catalog_manager.cc:1347] Generated new cluster ID: efbcbf215fc44d15945a2eabd1fe4ce7
I20260619 18:37:57.914605 5925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:37:57.922067 5925 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:37:57.923247 5925 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:37:57.928829 5925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 729f1e21f7fb4f05a5d32a2cdf08ee8a: Generated new TSK 0
I20260619 18:37:57.929603 5925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:37:58.019098 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:57.940799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34710 (local address 127.0.98.62:40643)
0619 18:37:57.940923 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:57.940928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:57.941092 (+ 164us) server_negotiation.cc:407] Connection header received
0619 18:37:57.941222 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:57.941225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:57.941283 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:57.941401 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0619 18:37:57.941568 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.942400 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.943216 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.943557 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.943651 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:57.943886 (+ 235us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 18:37:58.018641 (+ 74755us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 18:37:58.018945 (+ 304us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34710: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 18:37:58.019193 5914 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34710: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 18:37:58.019245 5940 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:37:57.940700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40643 (local address 127.0.0.1:34710)
0619 18:37:57.941015 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:57.941028 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:57.941118 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:57.941388 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:57.941390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:57.941396 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0619 18:37:57.941485 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:57.941489 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.942504 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.942507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:57.943121 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:57.943128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:57.943640 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:57.943656 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:57.943723 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:57.943744 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 18:37:58.018882 (+ 75138us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 18:37:58.019070 (+ 188us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40643: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":98,"threads_started":1}
I20260619 18:37:58.020025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 5844
2026-06-19T18:37:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (336 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (120 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (117 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:37:58 dist-test-slave-rxv6 krb5kdc[5952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:37:58 dist-test-slave-rxv6 krb5kdc[5952](info): set up 2 sockets
Jun 19 18:37:58 dist-test-slave-rxv6 krb5kdc[5952](info): commencing operation
krb5kdc: starting...
W20260619 18:37:59.205960 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:37:59 dist-test-slave-rxv6 krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:37:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:37:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:37:59.316928 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41153
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:59.424026 5965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:59.424317 5965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:59.424376 5965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:59.428468 5965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:37:59.428543 5965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:59.428570 5965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:59.428589 5965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:37:59.428609 5965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:37:59.433977 5965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:59.435038 5965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:59.436023 5965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:59.444087 5971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:59.444111 5970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:59.444111 5973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:59.444401 5965 server_base.cc:1034] running on GCE node
I20260619 18:37:59.444891 5965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:59.445940 5965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:59.447114 5965 hybrid_clock.cc:648] HybridClock initialized: now 1781894279447098 us; error 27 us; skew 500 ppm
Jun 19 18:37:59 dist-test-slave-rxv6 krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:59.449747 5965 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:37:59.450623 5965 webserver.cc:460] Webserver started at http://127.0.98.62:43839/ using document root <none> and password file <none>
I20260619 18:37:59.451161 5965 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:59.451225 5965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:59.451418 5965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:59.453394 5965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "897beaa16b8e46ce8400f944f7795d1a"
format_stamp: "Formatted at 2026-06-19 18:37:59 on dist-test-slave-rxv6"
server_key: "e14628022c22410cd8514f6c474b28e8"
server_key_iv: "7bb34de164725b0b0e0eb9532d10b283"
server_key_version: "encryptionkey@0"
I20260619 18:37:59.453967 5965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "897beaa16b8e46ce8400f944f7795d1a"
format_stamp: "Formatted at 2026-06-19 18:37:59 on dist-test-slave-rxv6"
server_key: "e14628022c22410cd8514f6c474b28e8"
server_key_iv: "7bb34de164725b0b0e0eb9532d10b283"
server_key_version: "encryptionkey@0"
I20260619 18:37:59.457489 5965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:37:59.459872 5980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:59.461210 5965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:37:59.461342 5965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "897beaa16b8e46ce8400f944f7795d1a"
format_stamp: "Formatted at 2026-06-19 18:37:59 on dist-test-slave-rxv6"
server_key: "e14628022c22410cd8514f6c474b28e8"
server_key_iv: "7bb34de164725b0b0e0eb9532d10b283"
server_key_version: "encryptionkey@0"
I20260619 18:37:59.461444 5965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:59.482551 5965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:59.485805 5965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:59.486037 5965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:59.492941 5965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41153
I20260619 18:37:59.493196 6032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41153 every 8 connection(s)
I20260619 18:37:59.493888 5965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:37:59.496556 6033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:37:59.502012 6033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a: Bootstrap starting.
I20260619 18:37:59.502776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 5965
I20260619 18:37:59.502897 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:37:59.503202 392 external_mini_cluster.cc:1428] Setting key cb6c022806086b26f27b65466d6102c2
I20260619 18:37:59.504861 6033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a: Neither blocks nor log segments found. Creating new log.
I20260619 18:37:59.505575 6033 log.cc:824] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:37:59.507746 6033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a: No bootstrap required, opened a new log
Jun 19 18:37:59 dist-test-slave-rxv6 krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:37:59.511912 6033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897beaa16b8e46ce8400f944f7795d1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41153 } }
I20260619 18:37:59.512161 6033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:37:59.512218 6033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897beaa16b8e46ce8400f944f7795d1a, State: Initialized, Role: FOLLOWER
I20260619 18:37:59.512669 6033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897beaa16b8e46ce8400f944f7795d1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41153 } }
I20260619 18:37:59.512805 6033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:37:59.512869 6033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:37:59.512966 6033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:37:59.513823 6033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897beaa16b8e46ce8400f944f7795d1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41153 } }
I20260619 18:37:59.514166 6033 leader_election.cc:302] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 897beaa16b8e46ce8400f944f7795d1a; no voters:
I20260619 18:37:59.514477 6033 leader_election.cc:288] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:37:59.514562 6038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:37:59.514760 6038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [term 1 LEADER]: Becoming Leader. State: Replica: 897beaa16b8e46ce8400f944f7795d1a, State: Running, Role: LEADER
I20260619 18:37:59.515110 6038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897beaa16b8e46ce8400f944f7795d1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41153 } }
I20260619 18:37:59.515900 6033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:37:59.516731 6040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 897beaa16b8e46ce8400f944f7795d1a. Latest consensus state: current_term: 1 leader_uuid: "897beaa16b8e46ce8400f944f7795d1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897beaa16b8e46ce8400f944f7795d1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41153 } } }
I20260619 18:37:59.517079 6040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:59.516991 6039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "897beaa16b8e46ce8400f944f7795d1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897beaa16b8e46ce8400f944f7795d1a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41153 } } }
I20260619 18:37:59.517419 6039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a [sys.catalog]: This master's current role is: LEADER
I20260619 18:37:59.517771 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:59.504357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.0.98.62:41153)
0619 18:37:59.504785 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:59.504804 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:59.504841 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:37:59.505507 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:59.505525 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:59.505898 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:59.506320 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:59.507150 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:59.508156 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:59.509293 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:59.509757 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:59.509868 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:59.512540 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:59.512545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:59.512558 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:59.512593 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:59.514226 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:59.514662 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:59.514668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:59.514674 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:59.514820 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:59.515155 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:59.515159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:59.515161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:59.515579 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:59.515786 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:59.516402 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 18:37:59.516720 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":143,"threads_started":1}
I20260619 18:37:59.518101 6047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:37:59.521337 6047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:37:59.526732 6047 catalog_manager.cc:1347] Generated new cluster ID: abfaf192026942f1b4483fa88ffd4935
I20260619 18:37:59.526809 6047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:37:59.547595 6047 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:37:59.548571 6047 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:37:59.577569 6047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 897beaa16b8e46ce8400f944f7795d1a: Generated new TSK 0
I20260619 18:37:59.578353 6047 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:37:59.613418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41153
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:59.736819 6059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:59.737098 6059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:59.737141 6059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:59.741207 6059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:59.741272 6059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:59.741330 6059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:37:59.746549 6059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:59.747521 6059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:59.748466 6059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:37:59.757810 6065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:59.757833 6067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:59.757810 6064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:59.758373 6059 server_base.cc:1034] running on GCE node
I20260619 18:37:59.758773 6059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:59.759393 6059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:59.760571 6059 hybrid_clock.cc:648] HybridClock initialized: now 1781894279760543 us; error 36 us; skew 500 ppm
Jun 19 18:37:59 dist-test-slave-rxv6 krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:59.763439 6059 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:37:59.764410 6059 webserver.cc:460] Webserver started at http://127.0.98.1:41041/ using document root <none> and password file <none>
I20260619 18:37:59.764946 6059 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:59.765015 6059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:59.765208 6059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:59.767300 6059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "c19d74d33dc24fab9a2c7788444a8ac8"
format_stamp: "Formatted at 2026-06-19 18:37:59 on dist-test-slave-rxv6"
server_key: "1f3f5f799c611f4675757d0ff663234a"
server_key_iv: "df1b3ecdac67079e3be9d1f4ba69a469"
server_key_version: "encryptionkey@0"
I20260619 18:37:59.767823 6059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c19d74d33dc24fab9a2c7788444a8ac8"
format_stamp: "Formatted at 2026-06-19 18:37:59 on dist-test-slave-rxv6"
server_key: "1f3f5f799c611f4675757d0ff663234a"
server_key_iv: "df1b3ecdac67079e3be9d1f4ba69a469"
server_key_version: "encryptionkey@0"
I20260619 18:37:59.771461 6059 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 18:37:59.773777 6074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:59.774951 6059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:37:59.775054 6059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "c19d74d33dc24fab9a2c7788444a8ac8"
format_stamp: "Formatted at 2026-06-19 18:37:59 on dist-test-slave-rxv6"
server_key: "1f3f5f799c611f4675757d0ff663234a"
server_key_iv: "df1b3ecdac67079e3be9d1f4ba69a469"
server_key_version: "encryptionkey@0"
I20260619 18:37:59.775159 6059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:59.784090 6059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:59.787067 6059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:59.787242 6059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:59.787781 6059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:59.788618 6059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:59.788666 6059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:59.788719 6059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:59.788748 6059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:59.798352 6059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34917
I20260619 18:37:59.798396 6187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34917 every 8 connection(s)
I20260619 18:37:59.799440 6059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:37:59.800079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 6059
I20260619 18:37:59.800293 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:37:59.800719 392 external_mini_cluster.cc:1428] Setting key 35157553b64b356c5f5f5725dc490960
Jun 19 18:37:59 dist-test-slave-rxv6 krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 18:37:59.816293 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:59.802907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58263 (local address 127.0.98.62:41153)
0619 18:37:59.803022 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 18:37:59.803036 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:37:59.803451 (+ 415us) server_negotiation.cc:407] Connection header received
0619 18:37:59.805177 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:37:59.805181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:37:59.805231 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:37:59.805314 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:37:59.806915 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:59.807730 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:59.809334 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:37:59.809662 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:37:59.809727 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:59.812637 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:37:59.812641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:37:59.812644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:37:59.812675 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:37:59.814211 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:59.814736 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:59.814738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:59.814740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:59.814810 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:37:59.815140 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:37:59.815144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:37:59.815146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:37:59.815362 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:37:59.815442 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 18:37:59.816059 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 18:37:59.816165 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:37:59.817544 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:37:59.801868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41153 (local address 127.0.98.1:58263)
0619 18:37:59.803303 (+ 1435us) negotiation.cc:107] Waiting for socket to connect
0619 18:37:59.803357 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 18:37:59.804942 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:37:59.805619 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:37:59.805628 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:37:59.806142 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:37:59.806759 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:59.806777 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:59.807876 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:59.807879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:59.809209 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:37:59.809219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:37:59.809933 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:37:59.809938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:37:59.810128 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:37:59.810694 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:37:59.810712 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:37:59.812431 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:37:59.814352 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:59.814357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:59.814370 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:59.814651 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:59.814870 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:37:59.814873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:37:59.814874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:37:59.815004 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:37:59.815500 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:37:59.815507 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:37:59.815824 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 18:37:59.816017 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 18:37:59.816394 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1224,"thread_start_us":134,"threads_started":1}
I20260619 18:37:59.818799 6188 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41153
I20260619 18:37:59.819031 6188 heartbeater.cc:464] Registering TS with master...
I20260619 18:37:59.819545 6188 heartbeater.cc:511] Master 127.0.98.62:41153 requested a full tablet report, sending...
I20260619 18:37:59.821022 5997 ts_manager.cc:194] Registered new tserver with Master: c19d74d33dc24fab9a2c7788444a8ac8 (127.0.98.1:34917)
I20260619 18:37:59.822166 5997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:37:59.830441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41153
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:37:59.936605 6193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:37:59.936885 6193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:37:59.936944 6193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:37:59.941006 6193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:37:59.941076 6193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:37:59.941159 6193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:37:59.946377 6193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:37:59.947404 6193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:37:59.948378 6193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:37:59.957597 6193 server_base.cc:1034] running on GCE node
W20260619 18:37:59.957562 6198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:59.957572 6201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:37:59.957824 6199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:37:59.958246 6193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:37:59.958844 6193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:37:59.959975 6193 hybrid_clock.cc:648] HybridClock initialized: now 1781894279959957 us; error 33 us; skew 500 ppm
Jun 19 18:37:59 dist-test-slave-rxv6 krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:37:59.962641 6193 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:37:59.963450 6193 webserver.cc:460] Webserver started at http://127.0.98.2:42435/ using document root <none> and password file <none>
I20260619 18:37:59.963982 6193 fs_manager.cc:359] Metadata directory not provided
I20260619 18:37:59.964042 6193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:37:59.964241 6193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:37:59.966375 6193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "694ed9ce930e4f5b885c31353c56e339"
format_stamp: "Formatted at 2026-06-19 18:37:59 on dist-test-slave-rxv6"
server_key: "22c71c2da584bddd2843978c78a59ef7"
server_key_iv: "11e0a5f7549b5190cc339fcee231c947"
server_key_version: "encryptionkey@0"
I20260619 18:37:59.966897 6193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "694ed9ce930e4f5b885c31353c56e339"
format_stamp: "Formatted at 2026-06-19 18:37:59 on dist-test-slave-rxv6"
server_key: "22c71c2da584bddd2843978c78a59ef7"
server_key_iv: "11e0a5f7549b5190cc339fcee231c947"
server_key_version: "encryptionkey@0"
I20260619 18:37:59.970372 6193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:37:59.972673 6208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:59.973814 6193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:37:59.973937 6193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "694ed9ce930e4f5b885c31353c56e339"
format_stamp: "Formatted at 2026-06-19 18:37:59 on dist-test-slave-rxv6"
server_key: "22c71c2da584bddd2843978c78a59ef7"
server_key_iv: "11e0a5f7549b5190cc339fcee231c947"
server_key_version: "encryptionkey@0"
I20260619 18:37:59.974032 6193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:37:59.993685 6193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:37:59.996399 6193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:37:59.996565 6193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:37:59.997125 6193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:37:59.997896 6193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:37:59.997960 6193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:37:59.998008 6193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:37:59.998039 6193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:00.005964 6193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45117
I20260619 18:38:00.006053 6321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45117 every 8 connection(s)
I20260619 18:38:00.006964 6193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 18:38:00 dist-test-slave-rxv6 krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:38:00.016288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 6193
I20260619 18:38:00.016398 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:38:00.016706 392 external_mini_cluster.cc:1428] Setting key 08ed36078fae97f70269bda6528fb4dd
I20260619 18:38:00.020458 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:00.008702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60399 (local address 127.0.98.62:41153)
0619 18:38:00.008864 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:00.008869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:00.009686 (+ 817us) server_negotiation.cc:407] Connection header received
0619 18:38:00.011012 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:00.011015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:00.011059 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:00.011129 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:00.012202 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:00.012714 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:00.013841 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:00.014190 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:00.014284 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:00.016904 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:00.016907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:00.016910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:00.016952 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:00.018519 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:00.018999 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:00.019002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:00.019004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:00.019079 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:00.019383 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:00.019386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:00.019388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:00.019573 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:00.019656 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:00.020178 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 18:38:00.020301 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:38:00.021279 6324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:00.009105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41153 (local address 127.0.98.2:60399)
0619 18:38:00.009546 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:00.009586 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:00.010832 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:00.011271 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:00.011277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:00.011605 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:00.012093 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:00.012107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:00.012796 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:00.012798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:00.013669 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:00.013677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:00.014339 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:00.014342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:00.014490 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:00.015062 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:00.015078 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:00.016707 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:00.018627 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:00.018632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:00.018649 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:00.018899 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:00.019158 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:00.019161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:00.019163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:00.019299 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:00.019667 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:00.019674 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:00.019953 (+ 279us) client_negotiation.cc:769] Sending connection context
0619 18:38:00.020124 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 18:38:00.020448 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":111,"threads_started":1}
I20260619 18:38:00.022562 6322 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41153
I20260619 18:38:00.022789 6322 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:00.023319 6322 heartbeater.cc:511] Master 127.0.98.62:41153 requested a full tablet report, sending...
I20260619 18:38:00.024400 5997 ts_manager.cc:194] Registered new tserver with Master: 694ed9ce930e4f5b885c31353c56e339 (127.0.98.2:45117)
I20260619 18:38:00.024981 5997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60399
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:38:00.041390 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41153
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:00.151150 6327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:00.151410 6327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:00.151459 6327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:00.155380 6327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:00.155431 6327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:00.155493 6327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:38:00.160754 6327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:00.161707 6327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:00.162712 6327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:00.172111 6333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:00.172101 6335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:00.172101 6332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:00.172430 6327 server_base.cc:1034] running on GCE node
I20260619 18:38:00.172870 6327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:00.173458 6327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:00.174614 6327 hybrid_clock.cc:648] HybridClock initialized: now 1781894280174593 us; error 33 us; skew 500 ppm
Jun 19 18:38:00 dist-test-slave-rxv6 krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:00.177134 6327 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:38:00.178045 6327 webserver.cc:460] Webserver started at http://127.0.98.3:37917/ using document root <none> and password file <none>
I20260619 18:38:00.178607 6327 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:00.178668 6327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:00.178889 6327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:00.180896 6327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "69e08ebce2d74e8bbdddbf207f46daae"
format_stamp: "Formatted at 2026-06-19 18:38:00 on dist-test-slave-rxv6"
server_key: "1dfe2012cc8dce340041dfc91633537d"
server_key_iv: "452041653a53132e43876fb053114a81"
server_key_version: "encryptionkey@0"
I20260619 18:38:00.181636 6327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "69e08ebce2d74e8bbdddbf207f46daae"
format_stamp: "Formatted at 2026-06-19 18:38:00 on dist-test-slave-rxv6"
server_key: "1dfe2012cc8dce340041dfc91633537d"
server_key_iv: "452041653a53132e43876fb053114a81"
server_key_version: "encryptionkey@0"
I20260619 18:38:00.185143 6327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:38:00.187722 6342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:00.188892 6327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:38:00.189033 6327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "69e08ebce2d74e8bbdddbf207f46daae"
format_stamp: "Formatted at 2026-06-19 18:38:00 on dist-test-slave-rxv6"
server_key: "1dfe2012cc8dce340041dfc91633537d"
server_key_iv: "452041653a53132e43876fb053114a81"
server_key_version: "encryptionkey@0"
I20260619 18:38:00.189136 6327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:00.206787 6327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:00.210105 6327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:00.210258 6327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:00.210780 6327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:00.211545 6327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:00.211591 6327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:00.211628 6327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:00.211642 6327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:00.220209 6327 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45835
I20260619 18:38:00.220243 6455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45835 every 8 connection(s)
I20260619 18:38:00.221061 6327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:38:00.227445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 6327
I20260619 18:38:00.227576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:38:00.227860 392 external_mini_cluster.cc:1428] Setting key 37d40a38e6a7e41e2a6bf5e33c197957
Jun 19 18:38:00 dist-test-slave-rxv6 krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:38:00.234127 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:00.222791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37581 (local address 127.0.98.62:41153)
0619 18:38:00.222925 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:00.222929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:00.223803 (+ 874us) server_negotiation.cc:407] Connection header received
0619 18:38:00.225075 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:00.225078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:00.225123 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:00.225194 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:00.226328 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:00.226997 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:00.228476 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:00.228713 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:00.228811 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:00.231202 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:00.231205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:00.231207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:00.231231 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:00.232365 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:00.232832 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:00.232835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:00.232836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:00.232915 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:00.233199 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:00.233202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:00.233203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:00.233384 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:00.233460 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:00.233870 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 18:38:00.234008 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:38:00.235026 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:00.223246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41153 (local address 127.0.98.3:37581)
0619 18:38:00.223664 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:00.223703 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:00.224895 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:00.225333 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:00.225339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:00.225684 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:00.226188 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:00.226201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:00.227157 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:00.227161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:00.228364 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:00.228374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:00.228834 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:00.228837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:00.229065 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:00.229607 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:00.229629 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:00.231071 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:00.232468 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:00.232472 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:00.232484 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:00.232730 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:00.232987 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:00.232989 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:00.232990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:00.233107 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:00.233457 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:00.233461 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:00.233692 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 18:38:00.233840 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 18:38:00.234124 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":109,"threads_started":1}
I20260619 18:38:00.236079 6456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41153
I20260619 18:38:00.236294 6456 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:00.236760 6456 heartbeater.cc:511] Master 127.0.98.62:41153 requested a full tablet report, sending...
I20260619 18:38:00.237715 5997 ts_manager.cc:194] Registered new tserver with Master: 69e08ebce2d74e8bbdddbf207f46daae (127.0.98.3:45835)
I20260619 18:38:00.238446 5997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37581
I20260619 18:38:00.241513 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/test-admin.keytab.
Jun 19 18:38:00 dist-test-slave-rxv6 krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:00.256393 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 18:38:00.259157 6463 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:00.257848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60112 (local address 127.0.98.1:34917)
0619 18:38:00.258183 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:00.258195 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:00.258217 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:38:00.258267 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:00.258272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:00.258454 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:00.258571 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:00.258989 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60112: BlockingRecv error: recv got EOF from 127.0.0.1:60112 (error 108)
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":147,"threads_started":1}
W20260619 18:38:00.259244 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 18:38:00 dist-test-slave-rxv6 krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:00.260301 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 18:38:00 dist-test-slave-rxv6 krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:00.261211 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 18:38:00 dist-test-slave-rxv6 krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 18:38:00.270138 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:00.262062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60120 (local address 127.0.98.1:34917)
0619 18:38:00.262171 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:00.262176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:00.262245 (+ 69us) server_negotiation.cc:407] Connection header received
0619 18:38:00.262372 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:00.262376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:00.262416 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:00.262476 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:00.262990 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:00.263769 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:00.264606 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:00.264977 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:00.265076 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:00.267145 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:00.267153 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:00.267167 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:00.267188 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:00.268506 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:00.268895 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:00.268900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:00.268901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:00.268972 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:00.269202 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:00.269206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:00.269207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:00.269440 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:00.269647 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:00.269815 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 18:38:00.269990 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 18:38:00.273674 6463 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:00.272862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.0.98.1:34917)
0619 18:38:00.272956 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:00.272958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:00.272991 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:38:00.273090 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:00.273092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:00.273125 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:00.273189 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:00.273576 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60124: BlockingRecv error: recv got EOF from 127.0.0.1:60124 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 18:38:00.274042 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/krb5cc)
Jun 19 18:38:00 dist-test-slave-rxv6 krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:00.274976 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 18:38:00 dist-test-slave-rxv6 krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:00.275779 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 18:38:00 dist-test-slave-rxv6 krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 18:38:00.283668 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:00.276513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60128 (local address 127.0.98.1:34917)
0619 18:38:00.276651 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:00.276655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:00.276751 (+ 96us) server_negotiation.cc:407] Connection header received
0619 18:38:00.276822 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:00.276824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:00.276861 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:00.276931 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:00.277337 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:00.278012 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:00.278867 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:00.279084 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:00.279165 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:00.281232 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:00.281235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:00.281237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:00.281258 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:00.282426 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:00.282769 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:00.282772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:00.282773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:00.282838 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:00.283014 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:00.283015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:00.283016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:00.283178 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:00.283269 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:00.283408 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 18:38:00.283493 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 18:38:00.286072 6463 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:00.285198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.0.98.1:34917)
0619 18:38:00.285310 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:00.285313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:00.285380 (+ 67us) server_negotiation.cc:407] Connection header received
0619 18:38:00.285492 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:00.285495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:00.285532 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:00.285593 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:00.285979 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60144: BlockingRecv error: recv got EOF from 127.0.0.1:60144 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 18:38:00.286144 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781894249862417-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 18:38:00 dist-test-slave-rxv6 krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:00.287002 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 18:38:00 dist-test-slave-rxv6 krb5kdc[5952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:00.287776 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 18:38:00 dist-test-slave-rxv6 krb5kdc[5952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 18:38:00.295727 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:00.288512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.0.98.1:34917)
0619 18:38:00.288614 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:00.288617 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:00.288703 (+ 86us) server_negotiation.cc:407] Connection header received
0619 18:38:00.288787 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:00.288789 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:00.288821 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:00.288885 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:00.289405 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:00.289876 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:00.290690 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:00.290892 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:00.290966 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:00.293005 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:00.293008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:00.293010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:00.293031 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:00.294321 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:00.294665 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:00.294668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:00.294669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:00.294742 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:00.295027 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:00.295029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:00.295030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:00.295203 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:00.295294 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:00.295520 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 18:38:00.295611 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:38:00.296643 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 6059
I20260619 18:38:00.301975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 6193
I20260619 18:38:00.306449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 6327
I20260619 18:38:00.310782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 5965
2026-06-19T18:38:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2053 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:38:00 dist-test-slave-rxv6 krb5kdc[6476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:38:00 dist-test-slave-rxv6 krb5kdc[6476](info): set up 2 sockets
Jun 19 18:38:00 dist-test-slave-rxv6 krb5kdc[6476](info): commencing operation
krb5kdc: starting...
W20260619 18:38:01.270346 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:38:01 dist-test-slave-rxv6 krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:38:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:38:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 18:38:01.382974 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40467
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:01.488921 6489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:01.489163 6489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:01.489200 6489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:01.493379 6489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:38:01.493446 6489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:01.493465 6489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:01.493481 6489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:38:01.493520 6489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:38:01.498840 6489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:01.499822 6489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:01.500787 6489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:01.508962 6495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:01.509069 6494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:01.508989 6497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:01.509281 6489 server_base.cc:1034] running on GCE node
I20260619 18:38:01.509804 6489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:01.510951 6489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:01.512116 6489 hybrid_clock.cc:648] HybridClock initialized: now 1781894281512091 us; error 41 us; skew 500 ppm
Jun 19 18:38:01 dist-test-slave-rxv6 krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894281, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:01.519040 6489 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 18:38:01.519982 6489 webserver.cc:460] Webserver started at http://127.0.98.62:41933/ using document root <none> and password file <none>
I20260619 18:38:01.520517 6489 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:01.520576 6489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:01.520771 6489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:01.523025 6489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "8bed8bc0c05f47de8c4c6ea261714ba8"
format_stamp: "Formatted at 2026-06-19 18:38:01 on dist-test-slave-rxv6"
server_key: "67f3beaf2d288de8daa623bd1c9d58d7"
server_key_iv: "48092664fab45fc6bb896d3fa90dea52"
server_key_version: "encryptionkey@0"
I20260619 18:38:01.523617 6489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "8bed8bc0c05f47de8c4c6ea261714ba8"
format_stamp: "Formatted at 2026-06-19 18:38:01 on dist-test-slave-rxv6"
server_key: "67f3beaf2d288de8daa623bd1c9d58d7"
server_key_iv: "48092664fab45fc6bb896d3fa90dea52"
server_key_version: "encryptionkey@0"
I20260619 18:38:01.527488 6489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:38:01.529779 6504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:01.530975 6489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:38:01.531075 6489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "8bed8bc0c05f47de8c4c6ea261714ba8"
format_stamp: "Formatted at 2026-06-19 18:38:01 on dist-test-slave-rxv6"
server_key: "67f3beaf2d288de8daa623bd1c9d58d7"
server_key_iv: "48092664fab45fc6bb896d3fa90dea52"
server_key_version: "encryptionkey@0"
I20260619 18:38:01.531173 6489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:01.550832 6489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:01.553784 6489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:01.553978 6489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:01.560454 6489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40467
I20260619 18:38:01.560770 6556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40467 every 8 connection(s)
I20260619 18:38:01.561381 6489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:38:01.564312 6557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:01.568667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 6489
I20260619 18:38:01.568775 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:38:01.569036 392 external_mini_cluster.cc:1428] Setting key 4dd994850702a7c2f08c099736b772fd
I20260619 18:38:01.570290 6557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8: Bootstrap starting.
I20260619 18:38:01.572543 6557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:01.573278 6557 log.cc:824] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:01.575076 6557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8: No bootstrap required, opened a new log
Jun 19 18:38:01 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 18:38:01.578794 6557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } }
I20260619 18:38:01.579010 6557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:01.579056 6557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bed8bc0c05f47de8c4c6ea261714ba8, State: Initialized, Role: FOLLOWER
I20260619 18:38:01.579478 6557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } }
I20260619 18:38:01.579608 6557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:38:01.579671 6557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:38:01.579766 6557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:01.580600 6557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } }
I20260619 18:38:01.580912 6557 leader_election.cc:302] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bed8bc0c05f47de8c4c6ea261714ba8; no voters:
I20260619 18:38:01.581212 6557 leader_election.cc:288] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:38:01.581300 6562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:38:01.581557 6562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [term 1 LEADER]: Becoming Leader. State: Replica: 8bed8bc0c05f47de8c4c6ea261714ba8, State: Running, Role: LEADER
I20260619 18:38:01.581952 6562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } }
I20260619 18:38:01.582588 6557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:38:01.582739 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:01.570453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60628 (local address 127.0.98.62:40467)
0619 18:38:01.570895 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:01.570910 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:01.570944 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:38:01.571558 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:01.571574 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:01.571917 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:01.572223 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:01.573006 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:01.574050 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:01.574854 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:01.575378 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:01.575519 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:01.578090 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:01.578095 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:01.578109 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:01.578135 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:01.579772 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:01.580156 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:01.580162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:01.580168 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:01.580273 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:01.580536 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:01.580539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:01.580541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:01.580965 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:01.581132 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:01.581430 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 18:38:01.581746 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":149,"threads_started":1}
I20260619 18:38:01.583509 6563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } }
I20260619 18:38:01.583627 6563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:01.583935 6564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bed8bc0c05f47de8c4c6ea261714ba8. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } }
I20260619 18:38:01.584009 6564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:01.585430 6571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:38:01.587986 6571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:38:01.593395 6571 catalog_manager.cc:1347] Generated new cluster ID: 7176c2da93324aad9f8aa24431e70a24
I20260619 18:38:01.593462 6571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:38:01.621395 6571 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:38:01.622725 6571 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:38:01.632153 6571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8: Generated new TSK 0
I20260619 18:38:01.632856 6571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 18:38:01.662623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40467
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:01.770893 6583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:01.771154 6583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:01.771214 6583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:01.775170 6583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:01.775231 6583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:01.775307 6583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:38:01.780462 6583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:01.781481 6583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:01.782521 6583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:01.792351 6591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:01.792385 6589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:01.792356 6588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:01.792407 6583 server_base.cc:1034] running on GCE node
I20260619 18:38:01.793299 6583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:01.794132 6583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:01.797430 6583 hybrid_clock.cc:648] HybridClock initialized: now 1781894281797386 us; error 55 us; skew 500 ppm
Jun 19 18:38:01 dist-test-slave-rxv6 krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894281, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:01.800455 6583 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 18:38:01.801339 6583 webserver.cc:460] Webserver started at http://127.0.98.1:38377/ using document root <none> and password file <none>
I20260619 18:38:01.801928 6583 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:01.802009 6583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:01.802210 6583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:01.804234 6583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "5effeb3931dd4e3189b71d2b8e2f40b5"
format_stamp: "Formatted at 2026-06-19 18:38:01 on dist-test-slave-rxv6"
server_key: "82abbad4a8e38ef021f31416e010d62f"
server_key_iv: "dfe9d5945bfe9e9973c9927480fa692e"
server_key_version: "encryptionkey@0"
I20260619 18:38:01.804795 6583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5effeb3931dd4e3189b71d2b8e2f40b5"
format_stamp: "Formatted at 2026-06-19 18:38:01 on dist-test-slave-rxv6"
server_key: "82abbad4a8e38ef021f31416e010d62f"
server_key_iv: "dfe9d5945bfe9e9973c9927480fa692e"
server_key_version: "encryptionkey@0"
I20260619 18:38:01.808519 6583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:38:01.810992 6598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:01.812214 6583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:38:01.812321 6583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "5effeb3931dd4e3189b71d2b8e2f40b5"
format_stamp: "Formatted at 2026-06-19 18:38:01 on dist-test-slave-rxv6"
server_key: "82abbad4a8e38ef021f31416e010d62f"
server_key_iv: "dfe9d5945bfe9e9973c9927480fa692e"
server_key_version: "encryptionkey@0"
I20260619 18:38:01.812431 6583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:01.831205 6583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:01.834390 6583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:01.834564 6583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:01.835150 6583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:01.835948 6583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:01.835994 6583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:01.836045 6583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:01.836073 6583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:01.844045 6583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42019
I20260619 18:38:01.844098 6711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42019 every 8 connection(s)
I20260619 18:38:01.844950 6583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:38:01.848853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 6583
I20260619 18:38:01.848976 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:38:01.849246 392 external_mini_cluster.cc:1428] Setting key a88190fe82c9a4da0bd93e3cca3afc05
Jun 19 18:38:01 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894281, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 18:38:01.858742 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:01.846750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55857 (local address 127.0.98.62:40467)
0619 18:38:01.846933 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:01.846938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:01.847743 (+ 805us) server_negotiation.cc:407] Connection header received
0619 18:38:01.849011 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:01.849015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:01.849070 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:01.849147 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:01.850428 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:01.851014 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:01.852151 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:01.852392 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:01.852482 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:01.855264 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:01.855277 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:01.855279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:01.855306 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:01.856701 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:01.857220 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:01.857223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:01.857224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:01.857298 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:01.857636 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:01.857639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:01.857641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:01.857837 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:01.857940 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:01.858498 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 18:38:01.858614 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 18:38:01.859772 6714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:01.847158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.1:55857)
0619 18:38:01.847600 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:01.847642 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:01.848799 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:01.849328 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:01.849335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:01.849726 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:01.850289 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:01.850307 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:01.851139 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:01.851142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:01.852033 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:01.852042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:01.852532 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:01.852536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:01.852706 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:01.853311 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:01.853327 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:01.855010 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:01.856831 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:01.856837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:01.856853 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:01.857115 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:01.857381 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:01.857384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:01.857386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:01.857544 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:01.857962 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:01.857970 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:01.858227 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 18:38:01.858455 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 18:38:01.858764 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260619 18:38:01.861084 6712 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40467
I20260619 18:38:01.861320 6712 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:01.861817 6712 heartbeater.cc:511] Master 127.0.98.62:40467 requested a full tablet report, sending...
I20260619 18:38:01.863270 6521 ts_manager.cc:194] Registered new tserver with Master: 5effeb3931dd4e3189b71d2b8e2f40b5 (127.0.98.1:42019)
I20260619 18:38:01.864457 6521 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 18:38:01.874814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40467
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:01.984871 6717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:01.985141 6717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:01.985190 6717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:01.989203 6717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:01.989276 6717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:01.989338 6717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:38:01.994835 6717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:01.995940 6717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:01.996968 6717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:02.007050 6722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:02.007138 6717 server_base.cc:1034] running on GCE node
W20260619 18:38:02.007045 6725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:02.007045 6723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:02.007714 6717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:02.008414 6717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:02.009590 6717 hybrid_clock.cc:648] HybridClock initialized: now 1781894282009568 us; error 34 us; skew 500 ppm
Jun 19 18:38:02 dist-test-slave-rxv6 krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:02.012490 6717 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 18:38:02.013480 6717 webserver.cc:460] Webserver started at http://127.0.98.2:42475/ using document root <none> and password file <none>
I20260619 18:38:02.014115 6717 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:02.014205 6717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:02.014416 6717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:02.016517 6717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "4a3e74f28b6c4b28996a66c22fd0bba4"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "8e04d8f077ab4a2405d036c84ba4fee1"
server_key_iv: "326030bd5402e0f6954a860ef356c690"
server_key_version: "encryptionkey@0"
I20260619 18:38:02.017181 6717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4a3e74f28b6c4b28996a66c22fd0bba4"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "8e04d8f077ab4a2405d036c84ba4fee1"
server_key_iv: "326030bd5402e0f6954a860ef356c690"
server_key_version: "encryptionkey@0"
I20260619 18:38:02.021044 6717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:38:02.023694 6732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:02.024996 6717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:38:02.025110 6717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "4a3e74f28b6c4b28996a66c22fd0bba4"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "8e04d8f077ab4a2405d036c84ba4fee1"
server_key_iv: "326030bd5402e0f6954a860ef356c690"
server_key_version: "encryptionkey@0"
I20260619 18:38:02.025209 6717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:02.040824 6717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:02.043740 6717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:02.043934 6717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:02.044595 6717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:02.045424 6717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:02.045480 6717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:02.045535 6717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:02.045562 6717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:02.053690 6717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35411
I20260619 18:38:02.053717 6845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35411 every 8 connection(s)
I20260619 18:38:02.054615 6717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:38:02.061197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 6717
I20260619 18:38:02.061314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:38:02.061573 392 external_mini_cluster.cc:1428] Setting key a42ef2da5d81600e2ffa1ce2618ed4cb
Jun 19 18:38:02 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 18:38:02.068447 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.056471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34143 (local address 127.0.98.62:40467)
0619 18:38:02.056607 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.056611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.057442 (+ 831us) server_negotiation.cc:407] Connection header received
0619 18:38:02.058751 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.058755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.058808 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.058879 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.060029 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.060537 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.061660 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.061871 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.062002 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.064567 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.064571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.064573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.064596 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.066109 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.066680 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.066683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.066684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.066770 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.067164 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.067167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.067168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.067408 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.067497 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.068157 (+ 660us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.068319 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:38:02.069517 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.056873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.2:34143)
0619 18:38:02.057298 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.057339 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:02.058547 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:02.059065 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:02.059072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:02.059425 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:02.059893 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.059907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.060651 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.060654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.061553 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.061562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.062038 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.062041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.062226 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.062851 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:02.062873 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:02.064400 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:02.066223 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.066230 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.066247 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.066592 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.066858 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.066861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.066864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.067053 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.067520 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:02.067527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:02.067873 (+ 346us) client_negotiation.cc:769] Sending connection context
0619 18:38:02.068105 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 18:38:02.068460 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 18:38:02.070745 6846 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40467
I20260619 18:38:02.070971 6846 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:02.071434 6846 heartbeater.cc:511] Master 127.0.98.62:40467 requested a full tablet report, sending...
I20260619 18:38:02.072458 6521 ts_manager.cc:194] Registered new tserver with Master: 4a3e74f28b6c4b28996a66c22fd0bba4 (127.0.98.2:35411)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260619 18:38:02.073155 6521 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 18:38:02.088006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40467
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:02.197168 6851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:02.197445 6851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:02.197489 6851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:02.201537 6851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:02.201617 6851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:02.201681 6851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:38:02.207212 6851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:02.208293 6851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:02.209296 6851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:02.218925 6857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:02.219002 6856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:02.218915 6859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:02.219130 6851 server_base.cc:1034] running on GCE node
I20260619 18:38:02.219573 6851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:02.220239 6851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:02.221418 6851 hybrid_clock.cc:648] HybridClock initialized: now 1781894282221396 us; error 33 us; skew 500 ppm
Jun 19 18:38:02 dist-test-slave-rxv6 krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:02.224151 6851 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 18:38:02.225078 6851 webserver.cc:460] Webserver started at http://127.0.98.3:37167/ using document root <none> and password file <none>
I20260619 18:38:02.225626 6851 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:02.225690 6851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:02.225875 6851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:02.227900 6851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "97a8dd9ef60d490f8ef3a7f44c47c099"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "813aeb371a066ac0eba3a81dea0e2199"
server_key_iv: "e9302ba22319e3fd02d1283e7f22d240"
server_key_version: "encryptionkey@0"
I20260619 18:38:02.228459 6851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "97a8dd9ef60d490f8ef3a7f44c47c099"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "813aeb371a066ac0eba3a81dea0e2199"
server_key_iv: "e9302ba22319e3fd02d1283e7f22d240"
server_key_version: "encryptionkey@0"
I20260619 18:38:02.232137 6851 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:38:02.234665 6866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:02.235977 6851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:38:02.236088 6851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "97a8dd9ef60d490f8ef3a7f44c47c099"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "813aeb371a066ac0eba3a81dea0e2199"
server_key_iv: "e9302ba22319e3fd02d1283e7f22d240"
server_key_version: "encryptionkey@0"
I20260619 18:38:02.236187 6851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:02.261636 6851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:02.264513 6851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:02.264689 6851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:02.265264 6851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:02.266100 6851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:02.266147 6851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:02.266196 6851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:02.266220 6851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:02.274330 6851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33855
I20260619 18:38:02.274376 6979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33855 every 8 connection(s)
I20260619 18:38:02.275175 6851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:38:02.284091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 6851
I20260619 18:38:02.284235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
Jun 19 18:38:02 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 18:38:02.284571 392 external_mini_cluster.cc:1428] Setting key ab10c11d302c40eac1898237c0240bb3
I20260619 18:38:02.288456 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.276903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53773 (local address 127.0.98.62:40467)
0619 18:38:02.277030 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.277033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.277823 (+ 790us) server_negotiation.cc:407] Connection header received
0619 18:38:02.279164 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.279168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.279223 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.279353 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.280432 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.280948 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.282100 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.282348 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.282435 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.285158 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.285161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.285164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.285192 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.286743 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.287228 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.287230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.287231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.287307 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.287558 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.287560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.287561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.287718 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.287804 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.288237 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.288341 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:38:02.289211 6982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.277259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.3:53773)
0619 18:38:02.277684 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.277725 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:02.278967 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:02.279483 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:02.279492 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:02.279843 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:02.280300 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.280313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.281060 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.281064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.281976 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.281985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.282493 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.282498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.282658 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.283297 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:02.283315 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:02.284988 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:02.286870 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.286876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.286891 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.287150 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.287370 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.287372 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.287373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.287497 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.287794 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:02.287799 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:02.288033 (+ 234us) client_negotiation.cc:769] Sending connection context
0619 18:38:02.288183 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 18:38:02.288420 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":112,"threads_started":1}
I20260619 18:38:02.290257 6980 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40467
I20260619 18:38:02.290479 6980 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:02.290946 6980 heartbeater.cc:511] Master 127.0.98.62:40467 requested a full tablet report, sending...
I20260619 18:38:02.291850 6521 ts_manager.cc:194] Registered new tserver with Master: 97a8dd9ef60d490f8ef3a7f44c47c099 (127.0.98.3:33855)
I20260619 18:38:02.292395 6521 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53773
I20260619 18:38:02.298691 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260619 18:38:02.322957 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44515
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40467,127.0.98.61:44515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:02.431284 6985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:02.431557 6985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:02.431602 6985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:02.435720 6985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:38:02.435787 6985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:02.435806 6985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:02.435822 6985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:38:02.435851 6985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:38:02.441233 6985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40467,127.0.98.61:44515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
W20260619 18:38:02.441458 6985 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:40467,127.0.98.61:44515'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 18:38:02.442400 6985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:02.443419 6985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:02.451918 6991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:02.452039 6985 server_base.cc:1034] running on GCE node
W20260619 18:38:02.451843 6993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:02.451838 6990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:02.452754 6985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:02.453859 6985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:02.455044 6985 hybrid_clock.cc:648] HybridClock initialized: now 1781894282455016 us; error 44 us; skew 500 ppm
Jun 19 18:38:02 dist-test-slave-rxv6 krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:02.457684 6985 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 18:38:02.458588 6985 webserver.cc:460] Webserver started at http://127.0.98.61:35377/ using document root <none> and password file <none>
I20260619 18:38:02.459131 6985 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:02.459190 6985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:02.459378 6985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:02.461352 6985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/data/instance:
uuid: "98fa3d886477428f995932b68902ee68"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "eb3c574f98c9cebd5e0e174ccd16132a"
server_key_iv: "d3ff57d0e275cd18d88fdcaf7f40a2f8"
server_key_version: "encryptionkey@0"
I20260619 18:38:02.461864 6985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/wal/instance:
uuid: "98fa3d886477428f995932b68902ee68"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "eb3c574f98c9cebd5e0e174ccd16132a"
server_key_iv: "d3ff57d0e275cd18d88fdcaf7f40a2f8"
server_key_version: "encryptionkey@0"
I20260619 18:38:02.465394 6985 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:38:02.467808 7000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:02.469054 6985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:38:02.469151 6985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/wal
uuid: "98fa3d886477428f995932b68902ee68"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "eb3c574f98c9cebd5e0e174ccd16132a"
server_key_iv: "d3ff57d0e275cd18d88fdcaf7f40a2f8"
server_key_version: "encryptionkey@0"
I20260619 18:38:02.469247 6985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:02.484316 6985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:02.487334 6985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:02.487498 6985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:02.493695 6985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44515
I20260619 18:38:02.493970 7052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44515 every 8 connection(s)
I20260619 18:38:02.494621 6985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/data/info.pb
I20260619 18:38:02.497231 7053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:02.498601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 6985
I20260619 18:38:02.501653 7053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:38:02 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 18:38:02 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 18:38:02.512665 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.503575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.0.98.62:40467)
0619 18:38:02.503712 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.503717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.503973 (+ 256us) server_negotiation.cc:407] Connection header received
0619 18:38:02.504117 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.504120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.504171 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.504232 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.505031 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.505820 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.507080 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.507283 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.507401 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.509605 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.509608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.509610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.509632 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.511021 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.511429 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.511431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.511433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.511496 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.511736 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.511739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.511741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.511997 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.512082 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.512444 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.512547 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:38:02.513274 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.500093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.0.98.61:44515)
0619 18:38:02.500500 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.500519 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.500562 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:38:02.501349 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.501372 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.501776 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.502163 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.502819 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.504011 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.505185 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.505880 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.506042 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.508621 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.508628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.508642 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.508667 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.510258 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.510689 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.510694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.510699 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.510795 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.511051 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.511053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.511054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.511553 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.511732 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.511972 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.512234 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":265,"thread_start_us":122,"threads_started":1}
I20260619 18:38:02.513274 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.503468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.0.1:60632)
0619 18:38:02.503857 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.503903 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:02.504023 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:02.504306 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:02.504312 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:02.504745 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:02.504915 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.504922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.506011 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.506014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.506928 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.506939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.507458 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.507462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.507569 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.508110 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:02.508125 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:02.509478 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:02.511109 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.511115 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.511121 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.511348 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.511570 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.511573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.511574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.511657 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.512089 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:02.512095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:02.512214 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 18:38:02.512350 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 18:38:02.512505 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":117,"threads_started":1}
I20260619 18:38:02.514628 7053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:38:02 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 18:38:02.521656 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.515125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44515 (local address 127.0.0.1:54760)
0619 18:38:02.515220 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.515233 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:02.515370 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:02.515621 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:02.515624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:02.515764 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:02.515859 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.515864 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.516928 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.516931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.517421 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.517428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.517805 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.517808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.517940 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.518299 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:02.518308 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:02.519359 (+ 1051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:02.520617 (+ 1258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.520620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.520621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.520783 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.520984 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.520986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.520987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.521046 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.521326 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:02.521327 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:02.521394 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 18:38:02.521462 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 18:38:02.521547 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":16}
I20260619 18:38:02.521821 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.515291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.0.98.61:44515)
0619 18:38:02.515399 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.515403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.515414 (+ 11us) server_negotiation.cc:407] Connection header received
0619 18:38:02.515492 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.515495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.515534 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.515600 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.515950 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.516839 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.517512 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.517723 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.517801 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.519470 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.519473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.519474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.519494 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.520526 (+ 1032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.520863 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.520865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.520866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.520922 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.521099 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.521100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.521102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.521263 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.521332 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.521565 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.521652 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":20}
I20260619 18:38:02.524485 7053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: Bootstrap starting.
I20260619 18:38:02.526767 7053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:02.527424 7053 log.cc:824] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:02.529277 7053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: No bootstrap required, opened a new log
I20260619 18:38:02.532150 7053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } }
I20260619 18:38:02.532337 7053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:02.532368 7053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98fa3d886477428f995932b68902ee68, State: Initialized, Role: FOLLOWER
I20260619 18:38:02.532717 7053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } }
I20260619 18:38:02.533113 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } } }
I20260619 18:38:02.533231 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:38:02.533846 7053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 18:38:02.538213 7070 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:38:02.538293 7070 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:38:02.540041 6985 master_runner.cc:428] Detected that this master 98fa3d886477428f995932b68902ee68 is joining an existing cluster
I20260619 18:38:02.540102 6985 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:44515
I20260619 18:38:02.549880 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.542323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.0.98.62:40467)
0619 18:38:02.542679 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.542683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.542739 (+ 56us) server_negotiation.cc:407] Connection header received
0619 18:38:02.542976 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.542980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.543026 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.543090 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.543656 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.544186 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.545049 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.545258 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.545352 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.546415 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.546419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.546422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.546446 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.548107 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.548624 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.548628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.548630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.548727 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.549019 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.549022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.549024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.549223 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.549305 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.549610 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.549728 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282}
I20260619 18:38:02.549881 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.542323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.0.1:60638)
0619 18:38:02.542661 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.542674 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:02.542868 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:02.543159 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:02.543163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:02.543426 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:02.543540 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.543546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.544293 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.544297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.544940 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.544948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.545368 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.545372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.545453 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.545953 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:02.545967 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:02.546281 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:02.548217 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.548220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.548222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.548506 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.548836 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.548839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.548841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.548926 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.549305 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:02.549308 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:02.549407 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 18:38:02.549546 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 18:38:02.549674 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":24,"thread_start_us":74,"threads_started":1}
I20260619 18:38:02.550246 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.542836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54770 (local address 127.0.98.61:44515)
0619 18:38:02.542946 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.542950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.542961 (+ 11us) server_negotiation.cc:407] Connection header received
0619 18:38:02.542991 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.542993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.543032 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.543099 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.543730 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.544526 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.545724 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.546069 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.546140 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.547372 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.547376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.547378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.547400 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.548769 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.549154 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.549156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.549158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.549233 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.549499 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.549503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.549504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.549683 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.549767 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.550058 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.550155 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:38:02.550614 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.542273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44515 (local address 127.0.0.1:54770)
0619 18:38:02.542567 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.542582 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:02.542679 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:02.543191 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:02.543194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:02.543474 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:02.543623 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.543631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.544647 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.544650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.545605 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.545615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.546278 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.546282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.546370 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.546809 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:02.546823 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:02.547208 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:02.548867 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.548870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.548872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.549067 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.549333 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.549335 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.549337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.549417 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.549796 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:02.549799 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:02.549868 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 18:38:02.550407 (+ 539us) client_negotiation.cc:240] Negotiation successful
0619 18:38:02.550497 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":16,"spinlock_wait_cycles":1408,"thread_start_us":120,"threads_started":1}
Jun 19 18:38:02 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894281, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 18:38:02.563036 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.554496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.0.98.61:44515)
0619 18:38:02.554603 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.554607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.554924 (+ 317us) server_negotiation.cc:407] Connection header received
0619 18:38:02.555106 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.555108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.555147 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.555202 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.555899 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.556370 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.557551 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.557744 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.557829 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.560161 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.560164 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.560166 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.560186 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.561402 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.561862 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.561865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.561867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.561968 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.562221 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.562224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.562225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.562378 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.562457 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.562767 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.562884 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:38:02.563043 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.554455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44515 (local address 127.0.0.1:54782)
0619 18:38:02.554795 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.554849 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:02.554975 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:02.555245 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:02.555251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:02.555683 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:02.555802 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.555809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.556456 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.556459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.557434 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.557444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.557837 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.557840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.557971 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.558448 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:02.558461 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:02.560040 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:02.561503 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.561511 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.561518 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.561749 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.562069 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.562072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.562073 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.562141 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.562474 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:02.562480 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:02.562592 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 18:38:02.562708 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 18:38:02.562889 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
I20260619 18:38:02.564512 6521 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:44515: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "98fa3d886477428f995932b68902ee68"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 44515
}
attrs {
promote: true
}
}
dest_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8"
cas_config_opid_index: -1
I20260619 18:38:02.565073 6521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true } }
I20260619 18:38:02.567909 7027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:02.569329 7027 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 1 FOLLOWER]: Refusing update from remote peer 8bed8bc0c05f47de8c4c6ea261714ba8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 18:38:02.570096 7082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:38:02.569876 7080 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 98fa3d886477428f995932b68902ee68 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true } } }
I20260619 18:38:02.575456 7027 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 98fa3d886477428f995932b68902ee68 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true } } }
I20260619 18:38:02.576459 7083 mvcc.cc:205] Tried to move back new op lower bound from 7298638977561456640 to 7298638977360539648. Current Snapshot: MvccSnapshot[applied={T|T < 7298638977397047296}]
I20260619 18:38:02.577270 7080 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:44515
I20260619 18:38:02.577561 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bed8bc0c05f47de8c4c6ea261714ba8. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true } } }
I20260619 18:38:02.577581 7082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true } } }
I20260619 18:38:02.577668 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: This master's current role is: LEARNER
I20260619 18:38:02.577683 7082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:02.577750 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true } } }
I20260619 18:38:02.577804 6985 master.cc:537] Master@127.0.98.61:44515 shutting down...
I20260619 18:38:02.577808 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: This master's current role is: LEARNER
I20260619 18:38:02.578074 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true } } }
I20260619 18:38:02.578135 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: This master's current role is: LEARNER
I20260619 18:38:02.582095 6985 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 1 LEARNER]: Raft consensus shutting down.
I20260619 18:38:02.582374 6985 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 1 LEARNER]: Raft consensus is shut down!
I20260619 18:38:02.582448 6985 tablet_replica.cc:330] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: stopping tablet replica
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260619 18:38:02.597429 6985 master.cc:552] Master@127.0.98.61:44515 shutdown complete.
I20260619 18:38:02.597482 6985 master_runner.cc:305] Clearing existing system tablet
I20260619 18:38:02.597714 6985 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:02.597771 6985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:02.598999 6985 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 18:38:02.599927 7093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:02.600116 6985 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:02.600235 6985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260619 18:38:02.600266 6985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/wal
uuid: "98fa3d886477428f995932b68902ee68"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "eb3c574f98c9cebd5e0e174ccd16132a"
server_key_iv: "d3ff57d0e275cd18d88fdcaf7f40a2f8"
server_key_version: "encryptionkey@0"
I20260619 18:38:02.601873 6985 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260619 18:38:02.604399 6985 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 18:38:02.604487 6985 log.cc:1196] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 18:38:02.604761 6985 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: Deleting consensus metadata
I20260619 18:38:02.604925 6985 master_runner.cc:315] Copying system tablet from 127.0.98.62:40467
I20260619 18:38:02.605118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43253
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40467,127.0.98.61:44515,127.0.98.60:43253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:38:02.606065 6985 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40467
I20260619 18:38:02.618703 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.606715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60648 (local address 127.0.98.62:40467)
0619 18:38:02.606833 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.606843 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.610327 (+ 3484us) server_negotiation.cc:407] Connection header received
0619 18:38:02.610474 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.610477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.610547 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.610645 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.611215 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.612048 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.613227 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.613579 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.613685 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.615027 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.615031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.615034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.615063 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.616976 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.617448 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.617452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.617454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.617542 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.617792 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.617796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.617797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.618054 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.618151 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.618405 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.618537 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:38:02.618716 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.606635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.0.1:60648)
0619 18:38:02.610182 (+ 3547us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.610200 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:02.610371 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:02.610680 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:02.610683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:02.610907 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:02.611080 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.611089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.612174 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.612178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.613099 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.613111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.613707 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.613710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.613814 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.614422 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:02.614437 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:02.614869 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:02.617080 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.617084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.617086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.617350 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.617623 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.617626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.617628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.617712 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.618138 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:02.618141 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:02.618244 (+ 103us) client_negotiation.cc:769] Sending connection context
0619 18:38:02.618351 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 18:38:02.618506 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3448,"thread_start_us":84,"threads_started":1}
I20260619 18:38:02.619819 6541 tablet_copy_service.cc:140] P 8bed8bc0c05f47de8c4c6ea261714ba8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 98fa3d886477428f995932b68902ee68 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60648)
I20260619 18:38:02.620005 6541 tablet_copy_service.cc:161] P 8bed8bc0c05f47de8c4c6ea261714ba8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 98fa3d886477428f995932b68902ee68 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60648: session id = 98fa3d886477428f995932b68902ee68-00000000000000000000000000000000
I20260619 18:38:02.621759 6541 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 18:38:02.623612 6985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 18:38:02.625988 6985 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: tablet copy: Starting download of 0 data blocks...
I20260619 18:38:02.626180 6985 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: tablet copy: Starting download of 1 WAL segments...
I20260619 18:38:02.628077 6985 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 18:38:02.629974 6541 tablet_copy_service.cc:342] P 8bed8bc0c05f47de8c4c6ea261714ba8: Request end of tablet copy session 98fa3d886477428f995932b68902ee68-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60648
I20260619 18:38:02.630110 6541 tablet_copy_service.cc:433] P 8bed8bc0c05f47de8c4c6ea261714ba8: ending tablet copy session 98fa3d886477428f995932b68902ee68-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 98fa3d886477428f995932b68902ee68
I20260619 18:38:02.637980 6985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:02.639787 7101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:02.639950 7102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:02.639787 7104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:02.642071 6985 server_base.cc:1034] running on GCE node
I20260619 18:38:02.642293 6985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:02.642592 6985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:02.643730 6985 hybrid_clock.cc:648] HybridClock initialized: now 1781894282643713 us; error 38 us; skew 500 ppm
Jun 19 18:38:02 dist-test-slave-rxv6 krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:02.645247 6985 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 18:38:02.645702 6985 webserver.cc:460] Webserver started at http://127.0.98.61:38915/ using document root <none> and password file <none>
I20260619 18:38:02.645884 6985 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:02.645963 6985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:02.647207 6985 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 18:38:02.648437 7111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:02.648689 6985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 18:38:02.648805 6985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/wal
uuid: "98fa3d886477428f995932b68902ee68"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "eb3c574f98c9cebd5e0e174ccd16132a"
server_key_iv: "d3ff57d0e275cd18d88fdcaf7f40a2f8"
server_key_version: "encryptionkey@0"
I20260619 18:38:02.648952 6985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:02.663048 6985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:02.665169 6985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:02.670776 6985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44515
I20260619 18:38:02.670806 7163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44515 every 8 connection(s)
I20260619 18:38:02.671598 6985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-1/data/info.pb
I20260619 18:38:02.673427 7164 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:38:02.674454 7164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: Bootstrap starting.
I20260619 18:38:02.681408 7164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:38:02.681697 7164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: Bootstrap complete.
I20260619 18:38:02.682281 7164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true } }
I20260619 18:38:02.682422 7164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 98fa3d886477428f995932b68902ee68, State: Initialized, Role: LEARNER
I20260619 18:38:02.682538 7164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true } }
I20260619 18:38:02.682838 7167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true } } }
I20260619 18:38:02.682962 7164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:38:02.683089 7167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: This master's current role is: LEARNER
I20260619 18:38:02.686494 7178 catalog_manager.cc:1259] Loaded cluster ID: 7176c2da93324aad9f8aa24431e70a24
I20260619 18:38:02.686553 7178 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: loading cluster ID for follower catalog manager: success
I20260619 18:38:02.688535 7178 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: acquiring CA information for follower catalog manager: success
I20260619 18:38:02.689801 7178 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 18:38:02.727491 7099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:02.727751 7099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:02.727805 7099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:02.731823 7099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:38:02.731909 7099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:02.731940 7099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:02.731969 7099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:38:02.731999 7099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:38:02.737255 7099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40467,127.0.98.61:44515,127.0.98.60:43253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:02.738320 7099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:02.739302 7099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:38:02.747440 7099 server_base.cc:1034] running on GCE node
W20260619 18:38:02.747400 7183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:02.747457 7186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:02.747400 7184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:02.748235 7099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:02.749373 7099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:02.750522 7099 hybrid_clock.cc:648] HybridClock initialized: now 1781894282750496 us; error 38 us; skew 500 ppm
Jun 19 18:38:02 dist-test-slave-rxv6 krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:02.753125 7099 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 18:38:02.753958 7099 webserver.cc:460] Webserver started at http://127.0.98.60:42749/ using document root <none> and password file <none>
I20260619 18:38:02.754513 7099 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:02.754573 7099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:02.754734 7099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:02.756652 7099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/data/instance:
uuid: "9d71d4874e74451180028c3b36607a98"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "52cbe2d412de146f98022fedfa5c06be"
server_key_iv: "6c3323431d496343aace0a84cbdce9b7"
server_key_version: "encryptionkey@0"
I20260619 18:38:02.757133 7099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/wal/instance:
uuid: "9d71d4874e74451180028c3b36607a98"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "52cbe2d412de146f98022fedfa5c06be"
server_key_iv: "6c3323431d496343aace0a84cbdce9b7"
server_key_version: "encryptionkey@0"
I20260619 18:38:02.760525 7099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 18:38:02.763031 7193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:02.764173 7099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:38:02.764267 7099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/wal
uuid: "9d71d4874e74451180028c3b36607a98"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "52cbe2d412de146f98022fedfa5c06be"
server_key_iv: "6c3323431d496343aace0a84cbdce9b7"
server_key_version: "encryptionkey@0"
I20260619 18:38:02.764377 7099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:02.778807 7099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:02.781766 7099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:02.781962 7099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:02.789018 7099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43253
I20260619 18:38:02.789247 7245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43253 every 8 connection(s)
I20260619 18:38:02.790094 7099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/data/info.pb
I20260619 18:38:02.792002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 7099
I20260619 18:38:02.793066 7246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:02.797796 7246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:38:02 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 18:38:02 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 18:38:02.806806 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.793829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38790 (local address 127.0.98.60:43253)
0619 18:38:02.794284 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.794298 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.794335 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:38:02.795417 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.795437 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.795772 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.796067 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.796935 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.797925 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.799052 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.799504 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.799642 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.802244 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.802251 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.802263 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.802294 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.803949 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.804354 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.804360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.804367 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.804496 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.804764 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.804767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.804769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.805209 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.805368 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.805655 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.805961 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":135,"threads_started":1}
I20260619 18:38:02.808979 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.799791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.0.98.62:40467)
0619 18:38:02.799906 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.799910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.800609 (+ 699us) server_negotiation.cc:407] Connection header received
0619 18:38:02.800722 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.800726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.800772 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.800837 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.801576 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.802154 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.803307 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.803548 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.803626 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.806019 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.806022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.806024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.806045 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.807469 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.807872 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.807875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.807876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.807942 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.808178 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.808181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.808183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.808338 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.808423 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.808712 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.808818 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:38:02.808992 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.799666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.0.1:57036)
0619 18:38:02.800456 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.800528 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:02.800636 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:02.800889 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:02.800895 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:02.801273 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:02.801429 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.801438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.802246 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.802249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.803200 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.803210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.803650 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.803654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.803760 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.804420 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:02.804433 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:02.805887 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:02.807565 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.807572 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.807578 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.807779 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.808021 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.808024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.808026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.808102 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.808431 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:02.808438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:02.808538 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 18:38:02.808651 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 18:38:02.808820 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":135,"threads_started":1}
I20260619 18:38:02.810241 7246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:38:02 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 18:38:02.817734 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.810671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44515 (local address 127.0.0.1:35278)
0619 18:38:02.810773 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.810786 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:02.810844 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:02.811171 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:02.811173 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:02.811292 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:02.811374 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.811378 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.812267 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.812269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.812810 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.812816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.813255 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.813257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.813320 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.813714 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:02.813724 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:02.814932 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:02.816422 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.816425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.816426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.816647 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.816909 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.816911 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.816912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.816975 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.817353 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:02.817355 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:02.817428 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 18:38:02.817525 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 18:38:02.817619 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 18:38:02.817777 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.810766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.0.98.61:44515)
0619 18:38:02.810996 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.810999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.811011 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:38:02.811044 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.811046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.811081 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.811150 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.811498 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.812184 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.812923 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.813167 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.813257 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.815074 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.815077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.815079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.815102 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.816325 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.816717 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.816719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.816720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.816782 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.817082 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.817085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.817087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.817274 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.817361 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.817564 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.817660 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":93,"threads_started":1}
I20260619 18:38:02.818459 7246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:38:02 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 18:38:02.825542 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.818895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43253 (local address 127.0.0.1:38802)
0619 18:38:02.819006 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.819019 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:02.819088 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:02.819262 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:02.819264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:02.819404 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:02.819460 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.819464 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.820472 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.820476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.820970 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.820977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.821350 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.821354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.821479 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.821830 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:02.821840 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:02.822953 (+ 1113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:02.824285 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.824288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.824290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.824485 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.824712 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.824714 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.824716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.824781 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.825093 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:02.825096 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:02.825165 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 18:38:02.825245 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 18:38:02.825369 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":30}
I20260619 18:38:02.825542 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.818953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.0.98.60:43253)
0619 18:38:02.819040 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.819044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.819056 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:38:02.819152 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.819155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.819188 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.819244 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.819551 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.820371 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.821063 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.821266 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.821353 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.823062 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.823066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.823068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.823088 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.824179 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.824566 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.824570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.824572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.824639 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.824861 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.824864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.824866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.825014 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.825092 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.825302 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.825418 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":36}
I20260619 18:38:02.828327 7246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: Bootstrap starting.
I20260619 18:38:02.830682 7246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:02.831385 7246 log.cc:824] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:02.833251 7246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: No bootstrap required, opened a new log
I20260619 18:38:02.836393 7246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } }
I20260619 18:38:02.836601 7246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:02.836661 7246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d71d4874e74451180028c3b36607a98, State: Initialized, Role: FOLLOWER
I20260619 18:38:02.837059 7246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } }
I20260619 18:38:02.837461 7252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } } }
I20260619 18:38:02.837582 7252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:38:02.838208 7246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 18:38:02.842481 7263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:38:02.842548 7263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:38:02.845697 7099 master_runner.cc:428] Detected that this master 9d71d4874e74451180028c3b36607a98 is joining an existing cluster
I20260619 18:38:02.845747 7099 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:43253
I20260619 18:38:02.856647 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.847753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57042 (local address 127.0.98.62:40467)
0619 18:38:02.847854 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.847859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.848909 (+ 1050us) server_negotiation.cc:407] Connection header received
0619 18:38:02.849028 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.849031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.849083 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.849159 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.849765 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.850625 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.852283 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.852643 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.852719 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.853947 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.853952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.853955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.853981 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.855192 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.855631 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.855633 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.855634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.855707 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.855958 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.855960 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.855961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.856121 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.856191 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.856436 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.856527 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:38:02.856714 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.847960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.0.1:57042)
0619 18:38:02.848842 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.848854 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:02.848942 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:02.849193 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:02.849196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:02.849446 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:02.849621 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.849630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.850766 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.850769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.852146 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.852157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.852869 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.852873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.852965 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.853349 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:02.853363 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:02.853777 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:02.855292 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.855296 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.855298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.855545 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.855782 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.855785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.855787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.855875 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.856196 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:02.856198 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:02.856286 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 18:38:02.856396 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 18:38:02.856527 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":419,"thread_start_us":80,"threads_started":1}
I20260619 18:38:02.860231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 6583
Jun 19 18:38:02 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894281, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 18:38:02.868453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42019
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40467
--webserver_port=38377
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40467,127.0.98.61:44515
--tserver_master_addrs=127.0.98.62:40467,127.0.98.61:44515,127.0.98.60:43253 with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:38:02.875034 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.848528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43253 (local address 127.0.0.1:38814)
0619 18:38:02.849155 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.849167 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:02.849259 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:02.849541 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:02.849544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:02.849706 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:02.849857 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.849865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.850797 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.850800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.851710 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.851719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.852352 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.852356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.852439 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.852951 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:02.852965 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:02.870984 (+ 18019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:02.873416 (+ 2432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.873419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.873421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.873769 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.874045 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.874048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.874050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.874124 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.874537 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:02.874539 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:02.874615 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 18:38:02.874700 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 18:38:02.874841 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":3,"thread_start_us":74,"threads_started":1}
I20260619 18:38:02.875198 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.847961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44515 (local address 127.0.0.1:35288)
0619 18:38:02.848534 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.848549 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:02.848678 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:02.849243 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:02.849246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:02.849528 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:02.849684 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.849691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.850844 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.850846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.851743 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.851749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.852377 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.852380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.852497 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.870378 (+ 17881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:02.870393 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:02.870971 (+ 578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:02.872926 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.872929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.872931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.873975 (+ 1044us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.874253 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.874256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.874258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.874334 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.874749 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:02.874752 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:02.874836 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 18:38:02.874924 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 18:38:02.875051 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":130,"spinlock_wait_cycles":9472,"thread_start_us":91,"threads_started":1}
I20260619 18:38:02.875525 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.848177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.0.98.60:43253)
0619 18:38:02.848273 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.848277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.849217 (+ 940us) server_negotiation.cc:407] Connection header received
0619 18:38:02.849351 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.849354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.849388 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.849431 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.849970 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.850684 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.851826 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.852126 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.852189 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.871118 (+ 18929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.871122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.871124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.871145 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.873315 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.873859 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.873862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.873864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.873962 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.874207 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.874209 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.874211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.874417 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.874492 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.875285 (+ 793us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.875385 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 18:38:02.875715 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.847731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.0.98.61:44515)
0619 18:38:02.847854 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.847859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.848611 (+ 752us) server_negotiation.cc:407] Connection header received
0619 18:38:02.848820 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.848824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.848889 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.848978 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.849821 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.850670 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.851861 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.852193 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.852253 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.871151 (+ 18898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.871155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.871158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.871191 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.872770 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.874086 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.874089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.874091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.874164 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.874428 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.874431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.874433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.874629 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.874713 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.875461 (+ 748us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.875578 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:38:02.877975 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.860237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.0.98.60:43253)
0619 18:38:02.862703 (+ 2466us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.862711 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.862722 (+ 11us) server_negotiation.cc:407] Connection header received
0619 18:38:02.862777 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.862779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.862818 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.862899 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.863308 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.863827 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.864591 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.864807 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.864888 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.866809 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.866812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.866814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.866836 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.873836 (+ 7000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.876859 (+ 3023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.876863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.876865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.876937 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.877189 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.877192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.877194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.877359 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.877440 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.877686 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.877783 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2407,"thread_start_us":92,"threads_started":1}
I20260619 18:38:02.877980 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.860200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43253 (local address 127.0.0.1:38824)
0619 18:38:02.860767 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.860788 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:02.860863 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:02.862936 (+ 2073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:02.862940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:02.863125 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:02.863206 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.863212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.863925 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.863928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.864492 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.864500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.864899 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.864903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.864993 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.865402 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:02.865415 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:02.866665 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:02.876530 (+ 9865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.876533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.876535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.876769 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.877024 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.877027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.877029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.877108 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.877450 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:02.877453 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:02.877546 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 18:38:02.877631 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 18:38:02.877786 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485}
I20260619 18:38:02.878890 6521 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:43253: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9d71d4874e74451180028c3b36607a98"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 43253
}
attrs {
promote: true
}
}
dest_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8"
cas_config_opid_index: 5
I20260619 18:38:02.879437 6521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true } }
I20260619 18:38:02.880421 7079 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9d71d4874e74451180028c3b36607a98 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true } } }
I20260619 18:38:02.881037 7220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:02.881850 7079 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:43253
I20260619 18:38:02.882055 7079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true } } }
I20260619 18:38:02.882160 7079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:02.882179 7220 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 1 FOLLOWER]: Refusing update from remote peer 8bed8bc0c05f47de8c4c6ea261714ba8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 18:38:02.882628 7099 master.cc:537] Master@127.0.98.60:43253 shutting down...
I20260619 18:38:02.882727 7082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 18:38:02.883992 6508 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 18:38:02.887071 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.880580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44515 (local address 127.0.0.1:35304)
0619 18:38:02.882621 (+ 2041us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.882635 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:02.882711 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:02.883136 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:02.883139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:02.883157 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:02.883306 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.883314 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.884453 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.884456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.885800 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.885813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.886649 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.886653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.886737 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.886780 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 18:38:02.886858 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 18:38:02.886931 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1968}
I20260619 18:38:02.887254 7099 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:38:02.887559 7099 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:02.887673 7099 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: stopping tablet replica
I20260619 18:38:02.888170 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.880788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.0.98.61:44515)
0619 18:38:02.880970 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.880974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.882791 (+ 1817us) server_negotiation.cc:407] Connection header received
0619 18:38:02.882839 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.882842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.882902 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.883004 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:02.883440 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.884265 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.885955 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.886541 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.886635 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.886788 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.886871 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.886934 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260619 18:38:02.888628 7138 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 1 LEARNER]: Refusing update from remote peer 8bed8bc0c05f47de8c4c6ea261714ba8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 18:38:02.889070 7082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 18:38:02.890421 7138 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9d71d4874e74451180028c3b36607a98 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true } } }
I20260619 18:38:02.892086 7167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bed8bc0c05f47de8c4c6ea261714ba8. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true } } }
I20260619 18:38:02.892189 7167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: This master's current role is: LEARNER
I20260619 18:38:02.892277 7167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: true } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true } } }
I20260619 18:38:02.892339 7167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: This master's current role is: LEARNER
I20260619 18:38:02.892331 7082 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8: attempting to promote NON_VOTER 98fa3d886477428f995932b68902ee68 to VOTER
I20260619 18:38:02.892750 7082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true } }
I20260619 18:38:02.894361 7138 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 1 LEARNER]: Refusing update from remote peer 8bed8bc0c05f47de8c4c6ea261714ba8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 18:38:02.894735 7080 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 18:38:02.895215 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.893833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43253 (local address 127.0.0.1:38828)
0619 18:38:02.895065 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.895138 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1155}
I20260619 18:38:02.896134 7079 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 98fa3d886477428f995932b68902ee68 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true } } }
I20260619 18:38:02.897272 7079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true } } }
I20260619 18:38:02.897372 7079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:02.897521 7079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true } } }
I20260619 18:38:02.897590 7079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:02.898085 7138 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 98fa3d886477428f995932b68902ee68 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true } } }
I20260619 18:38:02.898782 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.898340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43253 (local address 127.0.0.1:38838)
0619 18:38:02.898658 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.898711 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246}
W20260619 18:38:02.898969 6508 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 -> Peer 9d71d4874e74451180028c3b36607a98 (127.0.98.60:43253): Couldn't send request to peer 9d71d4874e74451180028c3b36607a98. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43253: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 18:38:02.899549 7276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true } } }
I20260619 18:38:02.899657 7276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:38:02.906026 7099 master.cc:552] Master@127.0.98.60:43253 shutdown complete.
I20260619 18:38:02.906085 7099 master_runner.cc:305] Clearing existing system tablet
I20260619 18:38:02.906299 7099 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:02.906356 7099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:02.907436 7099 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 18:38:02.908221 7278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:02.908448 7099 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:02.908578 7099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 18:38:02.908617 7099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/wal
uuid: "9d71d4874e74451180028c3b36607a98"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "52cbe2d412de146f98022fedfa5c06be"
server_key_iv: "6c3323431d496343aace0a84cbdce9b7"
server_key_version: "encryptionkey@0"
I20260619 18:38:02.910293 7099 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:02.912195 7099 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 18:38:02.912287 7099 log.cc:1196] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 18:38:02.912508 7099 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: Deleting consensus metadata
I20260619 18:38:02.912703 7099 master_runner.cc:315] Copying system tablet from 127.0.98.62:40467
I20260619 18:38:02.913760 7099 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40467
I20260619 18:38:02.921213 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.914419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57044 (local address 127.0.98.62:40467)
0619 18:38:02.914548 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:02.914553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:02.914757 (+ 204us) server_negotiation.cc:407] Connection header received
0619 18:38:02.914892 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:02.914895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:02.914942 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:02.915005 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:02.915444 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.916004 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.916992 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.917218 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.917306 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.918434 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:02.918438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:02.918441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:02.918467 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:02.919726 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.920111 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.920115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.920117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.920191 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:02.920442 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:02.920445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:02.920447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:02.920623 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:02.920706 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:02.920951 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 18:38:02.921053 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:38:02.921233 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:02.914330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.0.1:57044)
0619 18:38:02.914662 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:02.914674 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:02.914787 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:02.915033 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:02.915036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:02.915226 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:02.915324 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.915330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.916148 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.916151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.916873 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:02.916884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:02.917327 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:02.917331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:02.917423 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:02.917955 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:02.917970 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:02.918289 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:02.919831 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.919834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.919837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.920024 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.920270 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:02.920273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:02.920275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:02.920363 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:02.920709 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:02.920712 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:02.920804 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 18:38:02.920889 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 18:38:02.921048 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":82,"threads_started":1}
I20260619 18:38:02.921993 6541 tablet_copy_service.cc:140] P 8bed8bc0c05f47de8c4c6ea261714ba8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9d71d4874e74451180028c3b36607a98 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57044)
I20260619 18:38:02.922088 6541 tablet_copy_service.cc:161] P 8bed8bc0c05f47de8c4c6ea261714ba8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9d71d4874e74451180028c3b36607a98 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57044: session id = 9d71d4874e74451180028c3b36607a98-00000000000000000000000000000000
I20260619 18:38:02.922887 6541 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 18:38:02.924013 7099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 18:38:02.925827 7099 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: tablet copy: Starting download of 0 data blocks...
I20260619 18:38:02.926028 7099 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: tablet copy: Starting download of 1 WAL segments...
I20260619 18:38:02.927387 7099 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 18:38:02.928961 6541 tablet_copy_service.cc:342] P 8bed8bc0c05f47de8c4c6ea261714ba8: Request end of tablet copy session 9d71d4874e74451180028c3b36607a98-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57044
I20260619 18:38:02.929040 6541 tablet_copy_service.cc:433] P 8bed8bc0c05f47de8c4c6ea261714ba8: ending tablet copy session 9d71d4874e74451180028c3b36607a98-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9d71d4874e74451180028c3b36607a98
I20260619 18:38:02.935683 7099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:02.937290 7285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:02.937505 7288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:02.937546 7099 server_base.cc:1034] running on GCE node
W20260619 18:38:02.937638 7286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:02.937819 7099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:02.938117 7099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:02.939255 7099 hybrid_clock.cc:648] HybridClock initialized: now 1781894282939237 us; error 33 us; skew 500 ppm
Jun 19 18:38:02 dist-test-slave-rxv6 krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894282, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:02.940693 7099 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 18:38:02.941116 7099 webserver.cc:460] Webserver started at http://127.0.98.60:40771/ using document root <none> and password file <none>
I20260619 18:38:02.941289 7099 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:02.941331 7099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:02.942425 7099 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 18:38:02.943220 7295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:02.943437 7099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 18:38:02.943506 7099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/wal
uuid: "9d71d4874e74451180028c3b36607a98"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "52cbe2d412de146f98022fedfa5c06be"
server_key_iv: "6c3323431d496343aace0a84cbdce9b7"
server_key_version: "encryptionkey@0"
I20260619 18:38:02.943670 7099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:02.965158 7099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:02.967001 7099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:02.971287 7099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43253
I20260619 18:38:02.971297 7347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43253 every 8 connection(s)
I20260619 18:38:02.971794 7099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/master-2/data/info.pb
I20260619 18:38:02.973477 7348 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:38:02.974385 7348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: Bootstrap starting.
I20260619 18:38:02.982820 7348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:38:02.983094 7348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: Bootstrap complete.
I20260619 18:38:02.983702 7348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true } }
I20260619 18:38:02.983832 7348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d71d4874e74451180028c3b36607a98, State: Initialized, Role: LEARNER
I20260619 18:38:02.983937 7348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true } }
I20260619 18:38:02.984277 7351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true } } }
I20260619 18:38:02.984443 7348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:38:02.984521 7351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [sys.catalog]: This master's current role is: LEARNER
I20260619 18:38:02.987190 7362 catalog_manager.cc:1259] Loaded cluster ID: 7176c2da93324aad9f8aa24431e70a24
I20260619 18:38:02.987289 7362 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: loading cluster ID for follower catalog manager: success
I20260619 18:38:02.988863 7362 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: acquiring CA information for follower catalog manager: success
I20260619 18:38:02.989900 7362 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 18:38:03.021278 7273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:03.021564 7273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:03.021626 7273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:03.025761 7273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:03.025835 7273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:03.025942 7273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:38:03.031412 7273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38377
--tserver_master_addrs=127.0.98.62:40467,127.0.98.61:44515,127.0.98.60:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:03.032449 7273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:03.033743 7273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:03.043244 7370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:03.043259 7368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:03.043244 7367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:03.043571 7273 server_base.cc:1034] running on GCE node
I20260619 18:38:03.044106 7273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:03.044838 7273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:03.045969 7273 hybrid_clock.cc:648] HybridClock initialized: now 1781894283045951 us; error 26 us; skew 500 ppm
Jun 19 18:38:03 dist-test-slave-rxv6 krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:03.048682 7273 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 18:38:03.049571 7273 webserver.cc:460] Webserver started at http://127.0.98.1:38377/ using document root <none> and password file <none>
I20260619 18:38:03.050146 7273 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:03.050237 7273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:03.053830 7273 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:38:03.056028 7377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:03.057337 7273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:38:03.057437 7273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "5effeb3931dd4e3189b71d2b8e2f40b5"
format_stamp: "Formatted at 2026-06-19 18:38:01 on dist-test-slave-rxv6"
server_key: "82abbad4a8e38ef021f31416e010d62f"
server_key_iv: "dfe9d5945bfe9e9973c9927480fa692e"
server_key_version: "encryptionkey@0"
I20260619 18:38:03.057956 7273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:03.074172 7273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:03.075307 6846 heartbeater.cc:503] Master 127.0.98.62:40467 was elected leader, sending a full tablet report...
I20260619 18:38:03.077529 7273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:03.077700 7273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:03.078696 7273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:03.079543 7273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:03.079591 7273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:03.079644 7273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:03.079674 7273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:03.089071 7273 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42019
I20260619 18:38:03.089088 7490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42019 every 8 connection(s)
I20260619 18:38:03.090067 7273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:38:03.096210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 7273
I20260619 18:38:03.096390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 6717
Jun 19 18:38:03 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 18:38:03 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 18:38:03.103600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35411
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40467
--webserver_port=42475
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40467,127.0.98.61:44515
--tserver_master_addrs=127.0.98.62:40467,127.0.98.61:44515,127.0.98.60:43253 with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:38:03.105486 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.093597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45433 (local address 127.0.98.62:40467)
0619 18:38:03.093723 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.093727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.093811 (+ 84us) server_negotiation.cc:407] Connection header received
0619 18:38:03.094289 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.094292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.094336 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.094423 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:03.095868 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.096634 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.098303 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.098641 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.098716 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.101814 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:03.101818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:03.101820 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:03.101847 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:03.103405 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.103887 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.103890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.103892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.103970 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.104246 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.104250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.104252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.104468 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:03.104548 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.105187 (+ 639us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.105309 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:38:03.105491 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.091803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58195 (local address 127.0.98.60:43253)
0619 18:38:03.092028 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.092032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.093163 (+ 1131us) server_negotiation.cc:407] Connection header received
0619 18:38:03.094409 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.094412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.094469 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.094540 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:03.095848 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.098251 (+ 2403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.099164 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.099425 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.099526 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.101692 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:03.101696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:03.101699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:03.101726 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:03.103265 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.103786 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.103790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.103791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.103877 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.104199 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.104202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.104204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.104441 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:03.104541 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.105183 (+ 642us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.105321 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
I20260619 18:38:03.107149 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.092168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43253 (local address 127.0.98.1:58195)
0619 18:38:03.093017 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:03.093062 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:03.094016 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:03.094707 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:03.094713 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:03.095122 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:03.095715 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.095728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.098374 (+ 2646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.098378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.099054 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.099063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.099535 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.099539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.099631 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.100013 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:03.100025 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:03.101491 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:03.103364 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.103370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.103385 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.103666 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.103978 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.103980 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.103982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.104112 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.104545 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:03.104552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:03.104889 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 18:38:03.105648 (+ 759us) client_negotiation.cc:240] Negotiation successful
0619 18:38:03.106159 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":221,"spinlock_wait_cycles":13312,"thread_start_us":495,"threads_started":1}
I20260619 18:38:03.107153 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.093453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.1:45433)
0619 18:38:03.093722 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:03.093735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:03.094193 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:03.094708 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:03.094716 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:03.095129 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:03.095716 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.095732 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.097131 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.097134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.098166 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.098177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.098905 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.098908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.099109 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.099894 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:03.099914 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:03.101679 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:03.103522 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.103524 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.103526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.103775 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.104068 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.104071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.104073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.104147 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.104589 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:03.104592 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:03.104888 (+ 296us) client_negotiation.cc:769] Sending connection context
0619 18:38:03.105694 (+ 806us) client_negotiation.cc:240] Negotiation successful
0619 18:38:03.106086 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":137,"spinlock_wait_cycles":25728,"thread_start_us":124,"threads_started":1}
I20260619 18:38:03.108480 7491 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43253
I20260619 18:38:03.108486 7492 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40467
Jun 19 18:38:03 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 18:38:03.108726 7491 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:03.108727 7492 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:03.109354 7492 heartbeater.cc:511] Master 127.0.98.62:40467 requested a full tablet report, sending...
I20260619 18:38:03.109354 7491 heartbeater.cc:511] Master 127.0.98.60:43253 requested a full tablet report, sending...
I20260619 18:38:03.110841 6521 ts_manager.cc:194] Re-registered known tserver with Master: 5effeb3931dd4e3189b71d2b8e2f40b5 (127.0.98.1:42019)
I20260619 18:38:03.110914 7312 ts_manager.cc:194] Registered new tserver with Master: 5effeb3931dd4e3189b71d2b8e2f40b5 (127.0.98.1:42019)
I20260619 18:38:03.112613 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.093843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44515 (local address 127.0.98.1:43971)
0619 18:38:03.094100 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:03.094112 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:03.094200 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:03.094708 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:03.094716 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:03.095126 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:03.095716 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.095731 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.104955 (+ 9224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.104958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.105788 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.105797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.106555 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.106558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.106640 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.107067 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:03.107079 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:03.109017 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:03.110837 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.110840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.110842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.111061 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.111326 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.111328 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.111330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.111414 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.112105 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:03.112108 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:03.112216 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 18:38:03.112309 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 18:38:03.112442 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":132,"spinlock_wait_cycles":24960,"thread_start_us":113,"threads_started":1}
I20260619 18:38:03.112705 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.094107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43971 (local address 127.0.98.61:44515)
0619 18:38:03.094224 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.094227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.094243 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:38:03.094286 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.094289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.094337 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.094415 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:03.095882 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.096677 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.105950 (+ 9273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.106293 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.106384 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.109171 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:03.109175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:03.109178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:03.109205 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:03.110745 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.111143 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.111146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.111148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.111230 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.111789 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.111793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.111794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.112004 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:03.112096 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.112409 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.112569 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:38:03.113690 7493 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44515
I20260619 18:38:03.113773 7493 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:03.113847 6521 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45433
I20260619 18:38:03.113967 7493 heartbeater.cc:511] Master 127.0.98.61:44515 requested a full tablet report, sending...
I20260619 18:38:03.115079 7128 ts_manager.cc:194] Registered new tserver with Master: 5effeb3931dd4e3189b71d2b8e2f40b5 (127.0.98.1:42019)
W20260619 18:38:03.217458 7499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:03.217707 7499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:03.217749 7499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:03.221701 7499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:03.221768 7499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:03.221827 7499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:38:03.227166 7499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42475
--tserver_master_addrs=127.0.98.62:40467,127.0.98.61:44515,127.0.98.60:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:03.228243 7499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:03.229234 7499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:03.238857 7505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:03.238865 7504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:03.238878 7507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:03.239444 7499 server_base.cc:1034] running on GCE node
I20260619 18:38:03.240254 7499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:03.240942 7499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:03.242102 7499 hybrid_clock.cc:648] HybridClock initialized: now 1781894283242081 us; error 36 us; skew 500 ppm
Jun 19 18:38:03 dist-test-slave-rxv6 krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:03.245018 7499 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 18:38:03.245982 7499 webserver.cc:460] Webserver started at http://127.0.98.2:42475/ using document root <none> and password file <none>
I20260619 18:38:03.246541 7499 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:03.246620 7499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:03.250553 7499 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:38:03.252624 7514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:03.254034 7499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:38:03.254211 7499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "4a3e74f28b6c4b28996a66c22fd0bba4"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "8e04d8f077ab4a2405d036c84ba4fee1"
server_key_iv: "326030bd5402e0f6954a860ef356c690"
server_key_version: "encryptionkey@0"
I20260619 18:38:03.254770 7499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:03.269728 7499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:03.272619 7499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:03.272778 7499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:03.273324 7499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:03.274181 7499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:03.274245 7499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:03.274296 7499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:03.274319 7499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:03.282879 7499 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35411
I20260619 18:38:03.282917 7627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35411 every 8 connection(s)
I20260619 18:38:03.284123 7499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:38:03.291400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 7499
I20260619 18:38:03.291587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 6851
Jun 19 18:38:03 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 18:38:03 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 18:38:03 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 18:38:03.299729 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.286372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34641 (local address 127.0.98.62:40467)
0619 18:38:03.286522 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.286526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.287115 (+ 589us) server_negotiation.cc:407] Connection header received
0619 18:38:03.288154 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.288157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.288207 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.288257 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:03.289547 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.290345 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.291536 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.291864 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.291950 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.296102 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:03.296106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:03.296109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:03.296136 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:03.298003 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.298454 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.298457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.298459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.298540 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.298816 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.298819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.298821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.299025 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:03.299093 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.299456 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.299575 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:38:03.299729 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.285984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35235 (local address 127.0.98.61:44515)
0619 18:38:03.286114 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.286118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.287117 (+ 999us) server_negotiation.cc:407] Connection header received
0619 18:38:03.288125 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.288128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.288183 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.288254 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:03.289506 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.290338 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.291612 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.291942 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.292011 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.295636 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:03.295640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:03.295643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:03.295668 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:03.297314 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.297992 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.297996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.297998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.298086 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.298479 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.298482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.298484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.298715 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:03.298789 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.299448 (+ 659us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.299571 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:38:03.299921 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.286015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48303 (local address 127.0.98.60:43253)
0619 18:38:03.286152 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.286157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.287722 (+ 1565us) server_negotiation.cc:407] Connection header received
0619 18:38:03.288154 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.288157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.288209 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.288257 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:03.289529 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.290108 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.291611 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.291952 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.292026 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.296281 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:03.296284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:03.296287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:03.296321 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:03.298274 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.298709 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.298712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.298714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.298790 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.299058 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.299061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.299062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.299251 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:03.299316 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.299652 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.299769 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 18:38:03.299893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33855
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40467
--webserver_port=37167
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40467,127.0.98.61:44515
--tserver_master_addrs=127.0.98.62:40467,127.0.98.61:44515,127.0.98.60:43253 with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:38:03.301327 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.287391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43253 (local address 127.0.98.2:48303)
0619 18:38:03.287632 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:03.287643 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:03.287944 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:03.288424 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:03.288431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:03.288866 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:03.289384 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.289390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.290246 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.290248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.291497 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.291504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.292161 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.292165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.292404 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.293096 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:03.293115 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:03.296129 (+ 3014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:03.298376 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.298379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.298381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.298607 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.298886 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.298889 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.298891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.298965 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.299382 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:03.299385 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:03.299472 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 18:38:03.300090 (+ 618us) client_negotiation.cc:240] Negotiation successful
0619 18:38:03.300339 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":266,"spinlock_wait_cycles":19968,"thread_start_us":97,"threads_started":1}
I20260619 18:38:03.301333 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.286675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.2:34641)
0619 18:38:03.286971 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:03.287022 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:03.287939 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:03.288424 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:03.288431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:03.288853 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:03.289379 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.289388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.290486 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.290490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.291395 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.291406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.292141 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.292145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.292365 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.293078 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:03.293097 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:03.295947 (+ 2850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:03.298122 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.298125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.298127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.298340 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.298638 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.298641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.298643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.298722 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.299161 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:03.299164 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:03.299250 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 18:38:03.299898 (+ 648us) client_negotiation.cc:240] Negotiation successful
0619 18:38:03.300257 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":184,"spinlock_wait_cycles":22656,"thread_start_us":110,"threads_started":1}
I20260619 18:38:03.302199 7632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.286494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44515 (local address 127.0.98.2:35235)
0619 18:38:03.286971 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:03.287022 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:03.287937 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:03.288423 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:03.288429 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:03.288829 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:03.289372 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.289386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.290450 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.290453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.291476 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.291493 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.292161 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.292165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.292400 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.293195 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:03.293208 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:03.295446 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:03.297446 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.297452 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.297467 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.297826 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.298202 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.298205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.298207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.298379 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.298854 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:03.298860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:03.299192 (+ 332us) client_negotiation.cc:769] Sending connection context
0619 18:38:03.300532 (+ 1340us) client_negotiation.cc:240] Negotiation successful
0619 18:38:03.300660 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":173,"spinlock_wait_cycles":27648,"thread_start_us":104,"threads_started":1}
I20260619 18:38:03.302551 7628 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43253
I20260619 18:38:03.302587 7629 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40467
I20260619 18:38:03.302798 7629 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:03.302798 7628 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:03.303346 7629 heartbeater.cc:511] Master 127.0.98.62:40467 requested a full tablet report, sending...
I20260619 18:38:03.303346 7628 heartbeater.cc:511] Master 127.0.98.60:43253 requested a full tablet report, sending...
I20260619 18:38:03.303377 7630 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44515
I20260619 18:38:03.303507 7630 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:03.303629 7630 heartbeater.cc:511] Master 127.0.98.61:44515 requested a full tablet report, sending...
I20260619 18:38:03.304577 6521 ts_manager.cc:194] Re-registered known tserver with Master: 4a3e74f28b6c4b28996a66c22fd0bba4 (127.0.98.2:35411)
I20260619 18:38:03.304601 7128 ts_manager.cc:194] Registered new tserver with Master: 4a3e74f28b6c4b28996a66c22fd0bba4 (127.0.98.2:35411)
I20260619 18:38:03.304601 7312 ts_manager.cc:194] Registered new tserver with Master: 4a3e74f28b6c4b28996a66c22fd0bba4 (127.0.98.2:35411)
I20260619 18:38:03.305202 6521 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34641
W20260619 18:38:03.411140 7635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:03.411417 7635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:03.411487 7635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:03.415477 7635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:03.415547 7635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:03.415625 7635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:38:03.420907 7635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37167
--tserver_master_addrs=127.0.98.62:40467,127.0.98.61:44515,127.0.98.60:43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:03.422022 7635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:03.423048 7635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:03.432250 7640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:03.432250 7643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:03.432300 7641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:03.432473 7635 server_base.cc:1034] running on GCE node
I20260619 18:38:03.432917 7635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:03.433511 7635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:03.434671 7635 hybrid_clock.cc:648] HybridClock initialized: now 1781894283434650 us; error 37 us; skew 500 ppm
Jun 19 18:38:03 dist-test-slave-rxv6 krb5kdc[6476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:03.437635 7635 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 18:38:03.438580 7635 webserver.cc:460] Webserver started at http://127.0.98.3:37167/ using document root <none> and password file <none>
I20260619 18:38:03.439121 7635 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:03.439204 7635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:03.443056 7635 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 18:38:03.445238 7650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:03.446768 7635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:38:03.446869 7635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "97a8dd9ef60d490f8ef3a7f44c47c099"
format_stamp: "Formatted at 2026-06-19 18:38:02 on dist-test-slave-rxv6"
server_key: "813aeb371a066ac0eba3a81dea0e2199"
server_key_iv: "e9302ba22319e3fd02d1283e7f22d240"
server_key_version: "encryptionkey@0"
I20260619 18:38:03.447450 7635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:03.449405 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.445895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.0.98.60:43253)
0619 18:38:03.446026 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.446030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.446197 (+ 167us) server_negotiation.cc:407] Connection header received
0619 18:38:03.446331 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.446335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.446390 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.446471 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:03.446737 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.447278 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.448416 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.448872 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.448978 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.449121 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.449214 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.449280 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:38:03.449421 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.445824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43253 (local address 127.0.0.1:38840)
0619 18:38:03.446096 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:03.446109 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:03.446231 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:03.446496 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:03.446499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:03.446518 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:03.446640 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.446646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.447389 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.447445 (+ 56us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.448292 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.448303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.449003 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.449007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.449099 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.449122 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:38:03.449204 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 18:38:03.449280 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":101,"threads_started":1}
I20260619 18:38:03.449999 7322 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 18:38:03.450356 7351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bed8bc0c05f47de8c4c6ea261714ba8. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true } } }
I20260619 18:38:03.450485 7351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [sys.catalog]: This master's current role is: LEARNER
I20260619 18:38:03.469489 7635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:03.472379 7635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:03.472607 7635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:03.473239 7635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:03.474107 7635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:03.474156 7635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:03.474210 7635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:03.474234 7635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:03.482870 7635 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33855
I20260619 18:38:03.482924 7764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33855 every 8 connection(s)
I20260619 18:38:03.483994 7635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:38:03.487100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 7635
Jun 19 18:38:03 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 18:38:03 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 18:38:03 dist-test-slave-rxv6 krb5kdc[6476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894283, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 18:38:03.500717 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.485988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42229 (local address 127.0.98.60:43253)
0619 18:38:03.486214 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.486218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.487060 (+ 842us) server_negotiation.cc:407] Connection header received
0619 18:38:03.488263 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.488267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.488327 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.488379 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:03.490104 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.490817 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.492687 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.493029 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.493100 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.496899 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:03.496903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:03.496906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:03.496931 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:03.498458 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.499012 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.499016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.499018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.499111 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.499506 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.499509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.499511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.499718 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:03.499788 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.500450 (+ 662us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.500554 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260619 18:38:03.501622 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.485989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41817 (local address 127.0.98.61:44515)
0619 18:38:03.486117 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.486121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.489571 (+ 3450us) server_negotiation.cc:407] Connection header received
0619 18:38:03.489662 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.489665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.489712 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.489789 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:03.490306 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.491042 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.493980 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.494321 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.494385 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.497235 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:03.497239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:03.497242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:03.497266 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:03.499015 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.500336 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.500340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.500342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.500427 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.500738 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.500741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.500742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.500953 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:03.501032 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.501340 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.501472 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":51}
I20260619 18:38:03.502359 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.486381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43253 (local address 127.0.98.3:42229)
0619 18:38:03.486894 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:03.486951 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:03.488020 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:03.488562 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:03.488569 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:03.489157 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:03.489951 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.489971 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.490969 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.490972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.492556 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.492566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.493288 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.493292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.494132 (+ 840us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.494593 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:03.494608 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:03.496718 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:03.498571 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.498576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.498592 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.498899 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.499215 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.499218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.499220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.499396 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.499846 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:03.499852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:03.500161 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 18:38:03.500929 (+ 768us) client_negotiation.cc:240] Negotiation successful
0619 18:38:03.501289 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":891,"spinlock_wait_cycles":49408,"thread_start_us":125,"threads_started":1}
I20260619 18:38:03.502629 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.486381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44515 (local address 127.0.98.3:41817)
0619 18:38:03.489478 (+ 3097us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:03.489493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:03.489582 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:03.489892 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:03.489895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:03.490088 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:03.490188 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.490194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.491169 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.491172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.493837 (+ 2665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.493848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.495180 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.495183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.495247 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.495621 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:03.495634 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:03.497090 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:03.499098 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.499102 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.499104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.499322 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.500536 (+ 1214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.500539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.500541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.500632 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.501087 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:03.501090 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:03.501180 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 18:38:03.501414 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 18:38:03.501537 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":390,"thread_start_us":148,"threads_started":1}
I20260619 18:38:03.503988 7767 heartbeater.cc:347] Connected to a master server at 127.0.98.61:44515
I20260619 18:38:03.504032 7765 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43253
I20260619 18:38:03.504199 7767 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:03.504200 7765 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:03.504804 7765 heartbeater.cc:511] Master 127.0.98.60:43253 requested a full tablet report, sending...
I20260619 18:38:03.504804 7767 heartbeater.cc:511] Master 127.0.98.61:44515 requested a full tablet report, sending...
I20260619 18:38:03.505982 7312 ts_manager.cc:194] Registered new tserver with Master: 97a8dd9ef60d490f8ef3a7f44c47c099 (127.0.98.3:33855)
I20260619 18:38:03.505985 7128 ts_manager.cc:194] Registered new tserver with Master: 97a8dd9ef60d490f8ef3a7f44c47c099 (127.0.98.3:33855)
I20260619 18:38:03.506318 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.489876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38852 (local address 127.0.98.60:43253)
0619 18:38:03.490665 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.490668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.495038 (+ 4370us) server_negotiation.cc:407] Connection header received
0619 18:38:03.495206 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.495210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.495255 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.495311 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:03.496129 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.496660 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.498167 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.498509 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.498584 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.502828 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:03.502831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:03.502834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:03.502857 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:03.504434 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.504884 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.504887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.504889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.505007 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.505315 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.505318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.505319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.505525 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:03.505629 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.506052 (+ 423us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.506172 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":101,"threads_started":1}
I20260619 18:38:03.506958 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.488949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57060 (local address 127.0.98.62:40467)
0619 18:38:03.492396 (+ 3447us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.492400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.492413 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:38:03.492446 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.492449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.492489 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.492568 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:03.493393 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.494154 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.499678 (+ 5524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.499991 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.500075 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.501435 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:03.501439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:03.501441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:03.501464 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:03.503037 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.503384 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.503388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.503390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.503465 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.503757 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.503760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.503761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.503947 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:03.505659 (+ 1712us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.506324 (+ 665us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.506805 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3373,"thread_start_us":79,"threads_started":1}
I20260619 18:38:03.508635 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.489010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.0.98.61:44515)
0619 18:38:03.491051 (+ 2041us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.491057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.491074 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:38:03.491131 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.491135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.491185 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.491286 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:03.492226 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.492898 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.494052 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.494587 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.494669 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.498385 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:03.498389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:03.498392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:03.498418 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:03.505228 (+ 6810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.506976 (+ 1748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.506979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.506981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.507062 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.507344 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.507347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.507349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.507553 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:03.507665 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.508349 (+ 684us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.508472 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1970,"thread_start_us":90,"threads_started":1}
I20260619 18:38:03.508740 6560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.485780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33255 (local address 127.0.98.62:40467)
0619 18:38:03.485932 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.485936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.487760 (+ 1824us) server_negotiation.cc:407] Connection header received
0619 18:38:03.488247 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.488250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.488308 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.488380 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:03.490115 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.490894 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.492688 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.493029 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.493100 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.499292 (+ 6192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:03.499295 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:03.499298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:03.499324 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:03.501061 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.507561 (+ 6500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.507565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.507566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.507650 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.507944 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.507948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.507949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.508145 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:03.508229 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.508508 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.508610 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:38:03.508749 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.487384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.3:33255)
0619 18:38:03.487663 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:03.487675 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:03.488023 (+ 348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:03.488561 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:03.488568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:03.489160 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:03.489954 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.489972 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.491037 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.491040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.492556 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.492566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.493288 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.493292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.493536 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.494366 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:03.494386 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:03.497294 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:03.506938 (+ 9644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.506941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.506943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.507363 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.507751 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:03.507753 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:03.507755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:03.507838 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:03.508271 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:03.508274 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:03.508354 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 18:38:03.508430 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 18:38:03.508564 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":217,"spinlock_wait_cycles":16512,"thread_start_us":98,"threads_started":1}
I20260619 18:38:03.509266 7766 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40467
I20260619 18:38:03.509356 7766 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:03.509521 7766 heartbeater.cc:511] Master 127.0.98.62:40467 requested a full tablet report, sending...
I20260619 18:38:03.510133 6520 ts_manager.cc:194] Re-registered known tserver with Master: 97a8dd9ef60d490f8ef3a7f44c47c099 (127.0.98.3:33855)
I20260619 18:38:03.510665 6520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33255
I20260619 18:38:03.510183 6521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:38:03.513675 6521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:38:03.516374 7082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: true } } }
I20260619 18:38:03.516525 7082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:03.521692 7783 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8: attempting to promote NON_VOTER 9d71d4874e74451180028c3b36607a98 to VOTER
I20260619 18:38:03.522020 7783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false } }
I20260619 18:38:03.523398 7322 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 1 LEARNER]: Refusing update from remote peer 8bed8bc0c05f47de8c4c6ea261714ba8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260619 18:38:03.523406 7137 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 1 FOLLOWER]: Refusing update from remote peer 8bed8bc0c05f47de8c4c6ea261714ba8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260619 18:38:03.523769 7082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 18:38:03.524044 7783 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 18:38:03.525534 7082 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 9d71d4874e74451180028c3b36607a98 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false } } }
I20260619 18:38:03.526026 7137 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 9d71d4874e74451180028c3b36607a98 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false } } }
I20260619 18:38:03.526554 7322 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 9d71d4874e74451180028c3b36607a98 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false } } }
I20260619 18:38:03.526965 7082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false } } }
I20260619 18:38:03.527052 7082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:03.527166 7082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false } } }
I20260619 18:38:03.527228 7082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:03.527225 7784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false } } }
I20260619 18:38:03.527364 7784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:03.527585 7792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false } } }
I20260619 18:38:03.527762 7792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:38:03.528462 7351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false } } }
I20260619 18:38:03.528570 7351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:38:03.534660 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.529449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33855 (local address 127.0.0.1:39586)
0619 18:38:03.529563 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:03.529578 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:03.529673 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:03.530745 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:03.530747 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:03.530764 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:03.530863 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.530870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.532072 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.532075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.533142 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.533153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.534287 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.534291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.534379 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.534403 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:38:03.534464 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 18:38:03.534533 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 18:38:03.534729 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.529874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39586 (local address 127.0.98.3:33855)
0619 18:38:03.530223 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.530236 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.530260 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:38:03.530335 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.530341 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.530591 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.531012 (+ 421us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:03.531185 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.531967 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.533264 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.534172 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.534288 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.534445 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.534499 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.534588 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":119,"threads_started":1}
I20260619 18:38:03.535815 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.530219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42019 (local address 127.0.0.1:44480)
0619 18:38:03.530443 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:03.530456 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:03.530534 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:03.531715 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:03.531718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:03.531733 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:03.531820 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.531826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.532703 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.532707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.533704 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.533714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.535466 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.535470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.535554 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.535578 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:38:03.535624 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 18:38:03.535687 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260619 18:38:03.536444 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.530906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.0.98.1:42019)
0619 18:38:03.531295 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.531305 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.531329 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:38:03.531406 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.531424 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.531601 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.531725 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:03.531901 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.532587 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.534471 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.535308 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.536046 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.536158 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.536201 (+ 43us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.536297 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":86,"threads_started":1}
I20260619 18:38:03.536556 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.530500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35411 (local address 127.0.0.1:36094)
0619 18:38:03.531312 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:03.531323 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:03.531399 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:03.532644 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:03.532648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:03.532666 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:03.532791 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.532798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.533859 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.533862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.534900 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.534909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.536197 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.536201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.536282 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.536304 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:38:03.536368 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 18:38:03.536432 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":31,"thread_start_us":75,"threads_started":1}
I20260619 18:38:03.536976 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.531092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36094 (local address 127.0.98.2:35411)
0619 18:38:03.532239 (+ 1147us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.532250 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.532270 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:38:03.532338 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.532344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.532514 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.532651 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:03.532887 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.533774 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.535012 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.535892 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.536001 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.536138 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.536738 (+ 600us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.536829 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1050,"thread_start_us":95,"threads_started":1}
I20260619 18:38:03.537830 7699 tablet_service.cc:1467] Processing CreateTablet for tablet 9bedfe1b0b7447b897130bf8698c5afa (DEFAULT_TABLE table=test-table [id=a8c22b4a875c42e58dc4454dbc68d897]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:03.538822 7699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bedfe1b0b7447b897130bf8698c5afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:03.539083 7425 tablet_service.cc:1467] Processing CreateTablet for tablet 9bedfe1b0b7447b897130bf8698c5afa (DEFAULT_TABLE table=test-table [id=a8c22b4a875c42e58dc4454dbc68d897]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:03.539521 7562 tablet_service.cc:1467] Processing CreateTablet for tablet 9bedfe1b0b7447b897130bf8698c5afa (DEFAULT_TABLE table=test-table [id=a8c22b4a875c42e58dc4454dbc68d897]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:03.540074 7425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bedfe1b0b7447b897130bf8698c5afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:03.540436 7562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bedfe1b0b7447b897130bf8698c5afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:03.545375 7798 tablet_bootstrap.cc:492] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099: Bootstrap starting.
I20260619 18:38:03.547573 7798 tablet_bootstrap.cc:654] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:03.548290 7800 tablet_bootstrap.cc:492] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4: Bootstrap starting.
I20260619 18:38:03.548327 7798 log.cc:824] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:03.549614 7799 tablet_bootstrap.cc:492] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5: Bootstrap starting.
I20260619 18:38:03.550073 7798 tablet_bootstrap.cc:492] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099: No bootstrap required, opened a new log
I20260619 18:38:03.550269 7798 ts_tablet_manager.cc:1397] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:38:03.551399 7799 tablet_bootstrap.cc:654] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:03.551538 7800 tablet_bootstrap.cc:654] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:03.552125 7799 log.cc:824] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:03.552443 7800 log.cc:824] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:03.554536 7800 tablet_bootstrap.cc:492] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4: No bootstrap required, opened a new log
I20260619 18:38:03.554733 7800 ts_tablet_manager.cc:1397] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 18:38:03.554598 7798 raft_consensus.cc:348] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:03.554864 7798 raft_consensus.cc:374] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:03.554924 7798 raft_consensus.cc:729] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97a8dd9ef60d490f8ef3a7f44c47c099, State: Initialized, Role: FOLLOWER
I20260619 18:38:03.555372 7798 consensus_queue.cc:260] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:03.555549 7799 tablet_bootstrap.cc:492] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5: No bootstrap required, opened a new log
I20260619 18:38:03.555716 7799 ts_tablet_manager.cc:1397] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 18:38:03.556339 7798 ts_tablet_manager.cc:1428] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 18:38:03.556720 7766 heartbeater.cc:503] Master 127.0.98.62:40467 was elected leader, sending a full tablet report...
I20260619 18:38:03.559118 7800 raft_consensus.cc:348] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:03.559422 7800 raft_consensus.cc:374] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:03.559473 7800 raft_consensus.cc:729] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a3e74f28b6c4b28996a66c22fd0bba4, State: Initialized, Role: FOLLOWER
I20260619 18:38:03.559432 7799 raft_consensus.cc:348] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:03.559681 7799 raft_consensus.cc:374] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:03.559732 7799 raft_consensus.cc:729] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5effeb3931dd4e3189b71d2b8e2f40b5, State: Initialized, Role: FOLLOWER
I20260619 18:38:03.559893 7800 consensus_queue.cc:260] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:03.560153 7799 consensus_queue.cc:260] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:03.561170 7799 ts_tablet_manager.cc:1428] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260619 18:38:03.562206 7492 heartbeater.cc:503] Master 127.0.98.62:40467 was elected leader, sending a full tablet report...
I20260619 18:38:03.566638 7800 ts_tablet_manager.cc:1428] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4: Time spent starting tablet: real 0.012s user 0.001s sys 0.005s
I20260619 18:38:03.567039 7629 heartbeater.cc:503] Master 127.0.98.62:40467 was elected leader, sending a full tablet report...
W20260619 18:38:03.592011 7494 tablet.cc:2369] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:38:03.603031 7805 raft_consensus.cc:482] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:38:03.603237 7805 raft_consensus.cc:504] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:03.604432 7805 leader_election.cc:288] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97a8dd9ef60d490f8ef3a7f44c47c099 (127.0.98.3:33855), 5effeb3931dd4e3189b71d2b8e2f40b5 (127.0.98.1:42019)
I20260619 18:38:03.608032 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.604886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59533 (local address 127.0.98.1:42019)
0619 18:38:03.605004 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.605008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.605022 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:38:03.605071 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.605074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.605120 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.605180 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:03.605462 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.606008 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.607060 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.607470 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.607549 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.607603 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.607823 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.607890 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:38:03.608048 7634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.604674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42019 (local address 127.0.98.2:59533)
0619 18:38:03.604806 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:03.604822 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:03.604926 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:03.605205 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:03.605208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:03.605223 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:03.605354 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.605360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.606101 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.606104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.606905 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.606914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.607582 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.607586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.607683 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.607706 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:38:03.607775 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 18:38:03.607844 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 18:38:03.608201 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.604823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52847 (local address 127.0.98.3:33855)
0619 18:38:03.604956 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.604959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.605034 (+ 75us) server_negotiation.cc:407] Connection header received
0619 18:38:03.605142 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.605144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.605193 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.605265 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:03.605544 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.606096 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.607159 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.607556 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.607653 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.607713 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.608019 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.608096 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:38:03.608064 7632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.604674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33855 (local address 127.0.98.2:52847)
0619 18:38:03.604950 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:03.604964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:03.605054 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:03.605283 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:03.605285 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:03.605298 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:03.605390 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.605394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.606240 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.606243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.607047 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:03.607057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.607663 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.607666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:03.607735 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.607752 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 18:38:03.607822 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 18:38:03.607889 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":120}
I20260619 18:38:03.608891 7445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bedfe1b0b7447b897130bf8698c5afa" candidate_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" is_pre_election: true
I20260619 18:38:03.608904 7719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bedfe1b0b7447b897130bf8698c5afa" candidate_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" is_pre_election: true
I20260619 18:38:03.609205 7719 raft_consensus.cc:2454] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a3e74f28b6c4b28996a66c22fd0bba4 in term 0.
I20260619 18:38:03.609205 7445 raft_consensus.cc:2454] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a3e74f28b6c4b28996a66c22fd0bba4 in term 0.
I20260619 18:38:03.609652 7518 leader_election.cc:302] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4a3e74f28b6c4b28996a66c22fd0bba4, 5effeb3931dd4e3189b71d2b8e2f40b5, 97a8dd9ef60d490f8ef3a7f44c47c099; no voters:
I20260619 18:38:03.609994 7805 raft_consensus.cc:2781] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:38:03.610072 7805 raft_consensus.cc:482] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:38:03.610118 7805 raft_consensus.cc:3037] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:03.611212 7805 raft_consensus.cc:504] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:03.611560 7805 leader_election.cc:288] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [CANDIDATE]: Term 1 election: Requested vote from peers 97a8dd9ef60d490f8ef3a7f44c47c099 (127.0.98.3:33855), 5effeb3931dd4e3189b71d2b8e2f40b5 (127.0.98.1:42019)
I20260619 18:38:03.611860 7719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bedfe1b0b7447b897130bf8698c5afa" candidate_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099"
I20260619 18:38:03.611956 7445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bedfe1b0b7447b897130bf8698c5afa" candidate_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5"
I20260619 18:38:03.611972 7719 raft_consensus.cc:3037] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:03.612088 7445 raft_consensus.cc:3037] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:03.613106 7445 raft_consensus.cc:2454] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a3e74f28b6c4b28996a66c22fd0bba4 in term 1.
I20260619 18:38:03.613401 7518 leader_election.cc:302] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a3e74f28b6c4b28996a66c22fd0bba4, 5effeb3931dd4e3189b71d2b8e2f40b5; no voters:
I20260619 18:38:03.613570 7805 raft_consensus.cc:2781] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:38:03.613601 7719 raft_consensus.cc:2454] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a3e74f28b6c4b28996a66c22fd0bba4 in term 1.
I20260619 18:38:03.613775 7805 raft_consensus.cc:686] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 1 LEADER]: Becoming Leader. State: Replica: 4a3e74f28b6c4b28996a66c22fd0bba4, State: Running, Role: LEADER
I20260619 18:38:03.614157 7805 consensus_queue.cc:237] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:03.617048 6521 catalog_manager.cc:5495] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a3e74f28b6c4b28996a66c22fd0bba4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } health_report { overall_health: HEALTHY } } }
I20260619 18:38:03.636298 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.632717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36110 (local address 127.0.98.2:35411)
0619 18:38:03.632835 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.632839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.632853 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:38:03.632972 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.632975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.633029 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.633121 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:38:03.633364 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.633959 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.634838 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.635066 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.635153 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.635462 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:38:03.635920 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:38:03.636037 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.636099 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.636160 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:38:03.642278 7719 raft_consensus.cc:1261] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 1 FOLLOWER]: Refusing update from remote peer 4a3e74f28b6c4b28996a66c22fd0bba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:38:03.642278 7445 raft_consensus.cc:1261] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 1 FOLLOWER]: Refusing update from remote peer 4a3e74f28b6c4b28996a66c22fd0bba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:38:03.642944 7807 consensus_queue.cc:1035] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:38:03.643146 7805 consensus_queue.cc:1035] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:38:03.652220 7812 mvcc.cc:205] Tried to move back new op lower bound from 7298638985791225856 to 7298638985685954560. Current Snapshot: MvccSnapshot[applied={T|T < 7298638985791225856}]
I20260619 18:38:03.662763 6521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:38:03.662958 6521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:38:03.666965 6521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8bed8bc0c05f47de8c4c6ea261714ba8: Sending DeleteTablet for 3 replicas of tablet 9bedfe1b0b7447b897130bf8698c5afa
I20260619 18:38:03.667891 7699 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bedfe1b0b7447b897130bf8698c5afa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39586
I20260619 18:38:03.668092 7425 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bedfe1b0b7447b897130bf8698c5afa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44480
I20260619 18:38:03.668574 7821 tablet_replica.cc:330] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5: stopping tablet replica
I20260619 18:38:03.668869 7562 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bedfe1b0b7447b897130bf8698c5afa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36094
I20260619 18:38:03.669020 7821 raft_consensus.cc:2229] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:38:03.669289 7822 tablet_replica.cc:330] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4: stopping tablet replica
I20260619 18:38:03.669363 7821 raft_consensus.cc:2258] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:03.669672 7822 raft_consensus.cc:2229] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:38:03.670148 7822 raft_consensus.cc:2258] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:03.670979 7821 ts_tablet_manager.cc:1905] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:03.673521 7821 ts_tablet_manager.cc:1918] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:38:03.673622 7821 log.cc:1196] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-0/wal/wals/9bedfe1b0b7447b897130bf8698c5afa
I20260619 18:38:03.673900 7821 ts_tablet_manager.cc:1939] T 9bedfe1b0b7447b897130bf8698c5afa P 5effeb3931dd4e3189b71d2b8e2f40b5: Deleting consensus metadata
I20260619 18:38:03.675477 7822 ts_tablet_manager.cc:1905] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:03.677940 7822 ts_tablet_manager.cc:1918] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:38:03.678031 7822 log.cc:1196] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/9bedfe1b0b7447b897130bf8698c5afa
I20260619 18:38:03.678143 7820 tablet_replica.cc:330] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099: stopping tablet replica
I20260619 18:38:03.678195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 6489
I20260619 18:38:03.678321 7822 ts_tablet_manager.cc:1939] T 9bedfe1b0b7447b897130bf8698c5afa P 4a3e74f28b6c4b28996a66c22fd0bba4: Deleting consensus metadata
I20260619 18:38:03.678606 7820 raft_consensus.cc:2229] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:38:03.678927 7820 raft_consensus.cc:2258] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:03.680493 7820 ts_tablet_manager.cc:1905] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:03.682957 7820 ts_tablet_manager.cc:1918] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:38:03.683051 7820 log.cc:1196] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/9bedfe1b0b7447b897130bf8698c5afa
I20260619 18:38:03.683365 7820 ts_tablet_manager.cc:1939] T 9bedfe1b0b7447b897130bf8698c5afa P 97a8dd9ef60d490f8ef3a7f44c47c099: Deleting consensus metadata
I20260619 18:38:03.683918 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.670558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35316 (local address 127.0.98.61:44515)
0619 18:38:03.671050 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.671054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.674576 (+ 3522us) server_negotiation.cc:407] Connection header received
0619 18:38:03.674739 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.674742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.674785 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.674830 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:03.675814 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.676626 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.677792 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.678124 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.678206 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.680335 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:03.680339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:03.680342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:03.680367 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:03.682019 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.682484 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.682487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.682489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.682575 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.682839 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.682843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.682844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.683087 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:03.683211 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.683626 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.683763 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416}
W20260619 18:38:03.685460 7518 connection.cc:535] server connection from 127.0.0.1:36094 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 18:38:03.685505 7379 connection.cc:535] server connection from 127.0.0.1:44480 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 18:38:03.685874 7655 connection.cc:535] server connection from 127.0.0.1:39586 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 18:38:03.693488 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.670609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38866 (local address 127.0.98.60:43253)
0619 18:38:03.670715 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.670719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.684531 (+ 13812us) server_negotiation.cc:407] Connection header received
0619 18:38:03.684726 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.684729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.684788 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.684838 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:03.685716 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.686503 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.687634 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.687985 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.688055 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.690089 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:03.690093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:03.690095 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:03.690121 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:03.691708 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.692171 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.692174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.692176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.692255 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.692526 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.692529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.692531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.692732 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:03.692824 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.693142 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.693254 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 18:38:03.701808 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.689072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.0.98.60:43253)
0619 18:38:03.689171 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.689175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.693495 (+ 4320us) server_negotiation.cc:407] Connection header received
0619 18:38:03.693628 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.693631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.693675 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.693720 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:03.695025 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.695540 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.696630 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.696836 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.696919 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.698400 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:03.698403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:03.698406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:03.698428 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:03.699943 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.700303 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.700306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.700308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.700379 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.700614 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.700617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.700619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.700781 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:03.700896 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.701531 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.701642 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 18:38:03.705292 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:03.689689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.0.98.61:44515)
0619 18:38:03.689783 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:03.689786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:03.698172 (+ 8386us) server_negotiation.cc:407] Connection header received
0619 18:38:03.698277 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:03.698279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:03.698321 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:03.698393 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:03.699026 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.699730 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.700878 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:03.701132 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:03.701206 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:03.702704 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:03.702707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:03.702710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:03.702738 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:03.703883 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.704309 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.704312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.704314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.704393 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:03.704654 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:03.704656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:03.704657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:03.704808 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:03.704898 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:03.705090 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 18:38:03.705177 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 18:38:04.656762 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:04.656056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.2:53987)
0619 18:38:04.656376 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:04.656470 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":79,"threads_started":1}
W20260619 18:38:04.657053 7629 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40467 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
I20260619 18:38:04.657361 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:04.656227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.1:46499)
0619 18:38:04.657099 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:04.657269 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":71,"threads_started":1}
W20260619 18:38:04.657709 7492 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40467 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
I20260619 18:38:04.657964 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:04.656678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.3:36445)
0619 18:38:04.657310 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:04.657854 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":85,"threads_started":1}
W20260619 18:38:04.658175 7766 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40467 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
I20260619 18:38:05.168239 7834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8bed8bc0c05f47de8c4c6ea261714ba8)
I20260619 18:38:05.168426 7834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false } }
I20260619 18:38:05.168464 7835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8bed8bc0c05f47de8c4c6ea261714ba8)
I20260619 18:38:05.168627 7835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false } }
I20260619 18:38:05.169433 7834 leader_election.cc:288] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8bed8bc0c05f47de8c4c6ea261714ba8 (127.0.98.62:40467), 98fa3d886477428f995932b68902ee68 (127.0.98.61:44515)
I20260619 18:38:05.170066 7835 leader_election.cc:288] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8bed8bc0c05f47de8c4c6ea261714ba8 (127.0.98.62:40467), 9d71d4874e74451180028c3b36607a98 (127.0.98.60:43253)
I20260619 18:38:05.170105 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.169589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.0.1:57500)
0619 18:38:05.169884 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.169996 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":65,"threads_started":1}
W20260619 18:38:05.170377 7113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
I20260619 18:38:05.170572 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.169606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.0.1:57496)
0619 18:38:05.170338 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.170432 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":60,"threads_started":1}
W20260619 18:38:05.170814 7297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
I20260619 18:38:05.171682 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.171473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.0.1:57516)
0619 18:38:05.171587 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.171626 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 18:38:05.171701 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.171328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.0.1:57514)
0619 18:38:05.171567 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.171618 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260619 18:38:05.171792 7113 leader_election.cc:334] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8bed8bc0c05f47de8c4c6ea261714ba8 (127.0.98.62:40467): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
W20260619 18:38:05.171864 7297 leader_election.cc:334] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8bed8bc0c05f47de8c4c6ea261714ba8 (127.0.98.62:40467): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
I20260619 18:38:05.173851 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.169589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44515 (local address 127.0.0.1:35340)
0619 18:38:05.169867 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.169882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:05.170012 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:05.170525 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:05.170528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:05.170546 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:05.170661 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:05.170668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.171781 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.171784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:05.172656 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:05.172665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.173505 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.173507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:05.173577 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:05.173600 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:38:05.173666 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 18:38:05.173722 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":66,"threads_started":1}
I20260619 18:38:05.173892 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.169900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.0.98.61:44515)
0619 18:38:05.170284 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:05.170288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:05.170301 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:38:05.170349 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:05.170352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:05.170400 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:05.170773 (+ 373us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:05.170870 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.171646 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.172776 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.173385 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.173508 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:05.173574 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:05.173698 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 18:38:05.173771 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":292,"thread_start_us":65,"threads_started":1}
I20260619 18:38:05.174435 7137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d71d4874e74451180028c3b36607a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "98fa3d886477428f995932b68902ee68" is_pre_election: true
I20260619 18:38:05.174710 7137 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d71d4874e74451180028c3b36607a98 in term 1.
I20260619 18:38:05.175027 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.170420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43253 (local address 127.0.0.1:38882)
0619 18:38:05.170765 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.170778 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:05.170865 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:05.171474 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:05.171477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:05.171496 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:05.171580 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:05.171586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.172394 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.172396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:05.173660 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:05.173682 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.174615 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.174619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:05.174707 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:05.174728 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:38:05.174828 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 18:38:05.174901 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":32,"thread_start_us":82,"threads_started":1}
I20260619 18:38:05.175056 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.170169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.0.98.60:43253)
0619 18:38:05.170789 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:05.170816 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:05.170830 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:38:05.170978 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:05.170981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:05.171023 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:05.171077 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:05.171698 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.172286 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.173818 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.174497 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.174628 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:05.174684 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:05.174844 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 18:38:05.174949 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":85,"threads_started":1}
I20260619 18:38:05.175218 7298 leader_election.cc:302] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 98fa3d886477428f995932b68902ee68, 9d71d4874e74451180028c3b36607a98; no voters: 8bed8bc0c05f47de8c4c6ea261714ba8
I20260619 18:38:05.175447 7834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 18:38:05.175426 7322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98fa3d886477428f995932b68902ee68" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d71d4874e74451180028c3b36607a98" is_pre_election: true
I20260619 18:38:05.175501 7834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8bed8bc0c05f47de8c4c6ea261714ba8)
I20260619 18:38:05.175545 7834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:38:05.176504 7834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false } }
I20260619 18:38:05.176769 7322 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 98fa3d886477428f995932b68902ee68 in current term 2: Already voted for candidate 9d71d4874e74451180028c3b36607a98 in this term.
I20260619 18:38:05.176844 7834 leader_election.cc:288] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [CANDIDATE]: Term 2 election: Requested vote from peers 8bed8bc0c05f47de8c4c6ea261714ba8 (127.0.98.62:40467), 98fa3d886477428f995932b68902ee68 (127.0.98.61:44515)
I20260619 18:38:05.177220 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.176970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.0.1:57522)
0619 18:38:05.177128 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.177176 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 18:38:05.177193 7137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d71d4874e74451180028c3b36607a98" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "98fa3d886477428f995932b68902ee68"
I20260619 18:38:05.177309 7137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:38:05.177628 7115 leader_election.cc:302] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98fa3d886477428f995932b68902ee68; no voters: 8bed8bc0c05f47de8c4c6ea261714ba8, 9d71d4874e74451180028c3b36607a98
I20260619 18:38:05.178318 7137 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d71d4874e74451180028c3b36607a98 in term 2.
I20260619 18:38:05.178416 7835 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:38:05.178447 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.178261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.0.1:57538)
0619 18:38:05.178347 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.178392 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 18:38:05.178565 7298 leader_election.cc:302] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98fa3d886477428f995932b68902ee68, 9d71d4874e74451180028c3b36607a98; no voters:
I20260619 18:38:05.178689 7834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 2 FOLLOWER]: Leader election won for term 2
W20260619 18:38:05.178689 7297 leader_election.cc:334] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8bed8bc0c05f47de8c4c6ea261714ba8 (127.0.98.62:40467): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
I20260619 18:38:05.178910 7834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [term 2 LEADER]: Becoming Leader. State: Replica: 9d71d4874e74451180028c3b36607a98, State: Running, Role: LEADER
I20260619 18:38:05.179148 7834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false } }
I20260619 18:38:05.180166 7844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d71d4874e74451180028c3b36607a98. Latest consensus state: current_term: 2 leader_uuid: "9d71d4874e74451180028c3b36607a98" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false } } }
I20260619 18:38:05.180259 7844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:05.180516 7846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:38:05.182631 7846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:38:05.182994 7846 catalog_manager.cc:1259] Loaded cluster ID: 7176c2da93324aad9f8aa24431e70a24
I20260619 18:38:05.183040 7846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:38:05.183584 7846 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:38:05.183944 7846 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: Loaded TSK: 0
I20260619 18:38:05.184671 7846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:38:05.210817 7311 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:38:05.213800 7311 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:38:05.216521 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.216281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.0.1:57556)
0619 18:38:05.216390 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.216444 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 18:38:05.216660 7137 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [term 2 FOLLOWER]: Refusing update from remote peer 9d71d4874e74451180028c3b36607a98: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 18:38:05.217233 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.216959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.0.1:57562)
0619 18:38:05.217103 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.217166 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 18:38:05.217212 7834 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 18:38:05.217346 7297 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 -> Peer 8bed8bc0c05f47de8c4c6ea261714ba8 (127.0.98.62:40467): Couldn't send request to peer 8bed8bc0c05f47de8c4c6ea261714ba8. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 18:38:05.218573 7835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d71d4874e74451180028c3b36607a98. Latest consensus state: current_term: 2 leader_uuid: "9d71d4874e74451180028c3b36607a98" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false } } }
I20260619 18:38:05.218693 7835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:38:05.219614 7844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9d71d4874e74451180028c3b36607a98" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false } } }
I20260619 18:38:05.219725 7844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:05.220199 7835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9d71d4874e74451180028c3b36607a98" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8bed8bc0c05f47de8c4c6ea261714ba8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40467 } } peers { permanent_uuid: "98fa3d886477428f995932b68902ee68" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44515 } attrs { promote: false } } peers { permanent_uuid: "9d71d4874e74451180028c3b36607a98" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43253 } attrs { promote: false } } }
I20260619 18:38:05.220309 7835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fa3d886477428f995932b68902ee68 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:38:05.229585 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.225259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42019 (local address 127.0.0.1:44492)
0619 18:38:05.225676 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.225688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:05.225761 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:05.226357 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:05.226360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:05.226376 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:05.226472 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:05.226478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.227245 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.227248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:05.228384 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:05.228392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.229183 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.229186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:05.229271 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:05.229292 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:38:05.229363 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 18:38:05.229431 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354}
I20260619 18:38:05.229593 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.225934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44492 (local address 127.0.98.1:42019)
0619 18:38:05.226159 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:05.226164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:05.226176 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:38:05.226220 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:05.226223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:05.226266 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:05.226337 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:05.226570 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.227141 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.228504 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.229078 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.229172 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:05.229234 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:05.229386 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 18:38:05.229455 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":84,"threads_started":1}
I20260619 18:38:05.229630 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.225050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33855 (local address 127.0.0.1:39596)
0619 18:38:05.225189 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.225203 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:05.225277 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:05.226573 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:05.226576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:05.226597 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:05.226659 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:05.226663 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.227401 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.227403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:05.228228 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:05.228237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.229249 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.229252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:05.229339 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:05.229364 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:38:05.229408 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 18:38:05.229479 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"spinlock_wait_cycles":2304}
I20260619 18:38:05.230214 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.225456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39596 (local address 127.0.98.3:33855)
0619 18:38:05.226077 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:05.226082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:05.226096 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:38:05.226146 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:05.226148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:05.226206 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:05.226329 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:05.226763 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.227293 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.228487 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.229112 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.229889 (+ 777us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:05.229976 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:05.230012 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 18:38:05.230076 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":94,"threads_started":1}
I20260619 18:38:05.230176 7425 tablet_service.cc:1467] Processing CreateTablet for tablet 570543214a1d44ada34daa1a67ece96c (DEFAULT_TABLE table=test-table [id=3b4c495fc1974204b7468c6ce1afbaef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:05.230463 7425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 570543214a1d44ada34daa1a67ece96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:05.230732 7699 tablet_service.cc:1467] Processing CreateTablet for tablet 570543214a1d44ada34daa1a67ece96c (DEFAULT_TABLE table=test-table [id=3b4c495fc1974204b7468c6ce1afbaef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:05.230983 7699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 570543214a1d44ada34daa1a67ece96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:05.231065 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.225554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35411 (local address 127.0.0.1:36114)
0619 18:38:05.226181 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.226192 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:05.226277 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:05.226533 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:05.226536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:05.226554 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:05.226638 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:05.226644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.228288 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.228291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:05.229374 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:05.229380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.230222 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.230226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:05.230715 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:05.230738 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:38:05.230815 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 18:38:05.230930 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":74,"threads_started":1}
I20260619 18:38:05.231192 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.225864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36114 (local address 127.0.98.2:35411)
0619 18:38:05.226117 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:05.226122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:05.226256 (+ 134us) server_negotiation.cc:407] Connection header received
0619 18:38:05.226396 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:05.226399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:05.226444 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:05.226525 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:05.227446 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.228189 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.229488 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.230106 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.230256 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:05.230320 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:05.230879 (+ 559us) server_negotiation.cc:299] Negotiation successful
0619 18:38:05.230959 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":102,"threads_started":1}
I20260619 18:38:05.232851 7856 tablet_bootstrap.cc:492] T 570543214a1d44ada34daa1a67ece96c P 5effeb3931dd4e3189b71d2b8e2f40b5: Bootstrap starting.
I20260619 18:38:05.233297 7562 tablet_service.cc:1467] Processing CreateTablet for tablet 570543214a1d44ada34daa1a67ece96c (DEFAULT_TABLE table=test-table [id=3b4c495fc1974204b7468c6ce1afbaef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:05.233431 7857 tablet_bootstrap.cc:492] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099: Bootstrap starting.
I20260619 18:38:05.233548 7562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 570543214a1d44ada34daa1a67ece96c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:05.233950 7856 tablet_bootstrap.cc:654] T 570543214a1d44ada34daa1a67ece96c P 5effeb3931dd4e3189b71d2b8e2f40b5: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:05.234314 7857 tablet_bootstrap.cc:654] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:05.235208 7856 tablet_bootstrap.cc:492] T 570543214a1d44ada34daa1a67ece96c P 5effeb3931dd4e3189b71d2b8e2f40b5: No bootstrap required, opened a new log
I20260619 18:38:05.235286 7856 ts_tablet_manager.cc:1397] T 570543214a1d44ada34daa1a67ece96c P 5effeb3931dd4e3189b71d2b8e2f40b5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:38:05.235401 7857 tablet_bootstrap.cc:492] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099: No bootstrap required, opened a new log
I20260619 18:38:05.235493 7857 ts_tablet_manager.cc:1397] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:38:05.235737 7856 raft_consensus.cc:348] T 570543214a1d44ada34daa1a67ece96c P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:05.235862 7856 raft_consensus.cc:374] T 570543214a1d44ada34daa1a67ece96c P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:05.235894 7856 raft_consensus.cc:729] T 570543214a1d44ada34daa1a67ece96c P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5effeb3931dd4e3189b71d2b8e2f40b5, State: Initialized, Role: FOLLOWER
I20260619 18:38:05.235881 7857 raft_consensus.cc:348] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:05.235984 7857 raft_consensus.cc:374] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:05.236024 7857 raft_consensus.cc:729] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97a8dd9ef60d490f8ef3a7f44c47c099, State: Initialized, Role: FOLLOWER
I20260619 18:38:05.236006 7856 consensus_queue.cc:260] T 570543214a1d44ada34daa1a67ece96c P 5effeb3931dd4e3189b71d2b8e2f40b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:05.236172 7857 consensus_queue.cc:260] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:05.236306 7856 ts_tablet_manager.cc:1428] T 570543214a1d44ada34daa1a67ece96c P 5effeb3931dd4e3189b71d2b8e2f40b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:38:05.236498 7857 ts_tablet_manager.cc:1428] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:38:05.236554 7860 tablet_bootstrap.cc:492] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4: Bootstrap starting.
I20260619 18:38:05.237726 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.237289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.1:43257)
0619 18:38:05.237519 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.237621 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
W20260619 18:38:05.237865 7768 tablet.cc:2369] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:38:05.239275 7491 heartbeater.cc:503] Master 127.0.98.60:43253 was elected leader, sending a full tablet report...
I20260619 18:38:05.239604 7765 heartbeater.cc:503] Master 127.0.98.60:43253 was elected leader, sending a full tablet report...
I20260619 18:38:05.240415 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.237471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.3:40003)
0619 18:38:05.240256 (+ 2785us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.240314 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":83,"threads_started":1}
I20260619 18:38:05.240747 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.239647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.1:53877)
0619 18:38:05.240120 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.240685 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406}
I20260619 18:38:05.241153 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.240832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.3:52175)
0619 18:38:05.241033 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.241081 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
I20260619 18:38:05.241837 7860 tablet_bootstrap.cc:654] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:05.242865 7860 tablet_bootstrap.cc:492] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4: No bootstrap required, opened a new log
I20260619 18:38:05.242939 7860 ts_tablet_manager.cc:1397] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260619 18:38:05.243291 7860 raft_consensus.cc:348] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:05.243415 7860 raft_consensus.cc:374] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:05.243445 7860 raft_consensus.cc:729] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a3e74f28b6c4b28996a66c22fd0bba4, State: Initialized, Role: FOLLOWER
I20260619 18:38:05.243589 7860 consensus_queue.cc:260] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:05.243885 7860 ts_tablet_manager.cc:1428] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:38:05.244534 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.244147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.2:35075)
0619 18:38:05.244395 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.244448 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":100,"threads_started":1}
I20260619 18:38:05.245213 7628 heartbeater.cc:503] Master 127.0.98.60:43253 was elected leader, sending a full tablet report...
I20260619 18:38:05.245332 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.244938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.2:34645)
0619 18:38:05.245171 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.245210 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
I20260619 18:38:05.263687 7867 raft_consensus.cc:482] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:38:05.263801 7867 raft_consensus.cc:504] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:05.264180 7867 leader_election.cc:288] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97a8dd9ef60d490f8ef3a7f44c47c099 (127.0.98.3:33855), 5effeb3931dd4e3189b71d2b8e2f40b5 (127.0.98.1:42019)
I20260619 18:38:05.264560 7445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "570543214a1d44ada34daa1a67ece96c" candidate_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" is_pre_election: true
I20260619 18:38:05.264569 7719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "570543214a1d44ada34daa1a67ece96c" candidate_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" is_pre_election: true
I20260619 18:38:05.264708 7445 raft_consensus.cc:2454] T 570543214a1d44ada34daa1a67ece96c P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a3e74f28b6c4b28996a66c22fd0bba4 in term 0.
I20260619 18:38:05.264710 7719 raft_consensus.cc:2454] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a3e74f28b6c4b28996a66c22fd0bba4 in term 0.
I20260619 18:38:05.265009 7515 leader_election.cc:302] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a3e74f28b6c4b28996a66c22fd0bba4, 97a8dd9ef60d490f8ef3a7f44c47c099; no voters:
I20260619 18:38:05.265161 7867 raft_consensus.cc:2781] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:38:05.265218 7867 raft_consensus.cc:482] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:38:05.265244 7867 raft_consensus.cc:3037] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:05.266242 7867 raft_consensus.cc:504] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:05.266551 7867 leader_election.cc:288] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [CANDIDATE]: Term 1 election: Requested vote from peers 97a8dd9ef60d490f8ef3a7f44c47c099 (127.0.98.3:33855), 5effeb3931dd4e3189b71d2b8e2f40b5 (127.0.98.1:42019)
I20260619 18:38:05.266863 7445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "570543214a1d44ada34daa1a67ece96c" candidate_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5"
I20260619 18:38:05.266884 7719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "570543214a1d44ada34daa1a67ece96c" candidate_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099"
I20260619 18:38:05.266979 7719 raft_consensus.cc:3037] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:05.266997 7445 raft_consensus.cc:3037] T 570543214a1d44ada34daa1a67ece96c P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:05.267784 7719 raft_consensus.cc:2454] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a3e74f28b6c4b28996a66c22fd0bba4 in term 1.
I20260619 18:38:05.267865 7445 raft_consensus.cc:2454] T 570543214a1d44ada34daa1a67ece96c P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a3e74f28b6c4b28996a66c22fd0bba4 in term 1.
I20260619 18:38:05.268061 7515 leader_election.cc:302] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a3e74f28b6c4b28996a66c22fd0bba4, 97a8dd9ef60d490f8ef3a7f44c47c099; no voters:
I20260619 18:38:05.268260 7867 raft_consensus.cc:2781] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:38:05.268440 7867 raft_consensus.cc:686] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 1 LEADER]: Becoming Leader. State: Replica: 4a3e74f28b6c4b28996a66c22fd0bba4, State: Running, Role: LEADER
I20260619 18:38:05.268566 7867 consensus_queue.cc:237] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } }
I20260619 18:38:05.269176 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.268894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.2:39769)
0619 18:38:05.269046 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.269101 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 18:38:05.270316 7311 catalog_manager.cc:5495] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a3e74f28b6c4b28996a66c22fd0bba4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a3e74f28b6c4b28996a66c22fd0bba4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35411 } health_report { overall_health: HEALTHY } } }
W20260619 18:38:05.287276 7631 tablet.cc:2369] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:38:05.291867 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.289242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36116 (local address 127.0.98.2:35411)
0619 18:38:05.289343 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:05.289346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:05.289359 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:38:05.289438 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:05.289440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:05.289482 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:05.289541 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:38:05.289764 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.290302 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.291083 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:05.291296 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:05.291370 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:05.291550 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:38:05.291610 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:38:05.291655 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:05.291734 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 18:38:05.291776 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 18:38:05.293766 7445 raft_consensus.cc:1261] T 570543214a1d44ada34daa1a67ece96c P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 1 FOLLOWER]: Refusing update from remote peer 4a3e74f28b6c4b28996a66c22fd0bba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:38:05.293764 7719 raft_consensus.cc:1261] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 1 FOLLOWER]: Refusing update from remote peer 4a3e74f28b6c4b28996a66c22fd0bba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:38:05.294190 7870 consensus_queue.cc:1035] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a8dd9ef60d490f8ef3a7f44c47c099" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:38:05.294322 7867 consensus_queue.cc:1035] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5effeb3931dd4e3189b71d2b8e2f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:38:05.297377 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.297111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.3:42985)
0619 18:38:05.297220 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.297284 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 18:38:05.298305 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.296727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.1:48929)
0619 18:38:05.298137 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.298221 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1341}
I20260619 18:38:05.299131 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.298855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.2:51793)
0619 18:38:05.299008 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.299061 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 18:38:05.300534 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.300322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.1:45039)
0619 18:38:05.300413 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.300467 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 18:38:05.302723 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.302466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.2:59757)
0619 18:38:05.302593 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.302644 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 18:38:05.304348 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.302894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40467 (local address 127.0.98.3:52719)
0619 18:38:05.304219 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.304276 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1258}
I20260619 18:38:05.304556 7311 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:38:05.304723 7311 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:38:05.308241 7311 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d71d4874e74451180028c3b36607a98: Sending DeleteTablet for 3 replicas of tablet 570543214a1d44ada34daa1a67ece96c
I20260619 18:38:05.308905 7699 tablet_service.cc:1514] Processing DeleteTablet for tablet 570543214a1d44ada34daa1a67ece96c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:05 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39596
I20260619 18:38:05.309026 7425 tablet_service.cc:1514] Processing DeleteTablet for tablet 570543214a1d44ada34daa1a67ece96c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:05 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44492
I20260619 18:38:05.309203 7562 tablet_service.cc:1514] Processing DeleteTablet for tablet 570543214a1d44ada34daa1a67ece96c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:05 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36114
I20260619 18:38:05.309360 7884 tablet_replica.cc:330] T 570543214a1d44ada34daa1a67ece96c P 5effeb3931dd4e3189b71d2b8e2f40b5: stopping tablet replica
I20260619 18:38:05.309480 7884 raft_consensus.cc:2229] T 570543214a1d44ada34daa1a67ece96c P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:38:05.309530 7885 tablet_replica.cc:330] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4: stopping tablet replica
I20260619 18:38:05.309636 7884 raft_consensus.cc:2258] T 570543214a1d44ada34daa1a67ece96c P 5effeb3931dd4e3189b71d2b8e2f40b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:05.309643 7885 raft_consensus.cc:2229] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:38:05.309679 7883 tablet_replica.cc:330] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099: stopping tablet replica
I20260619 18:38:05.309773 7883 raft_consensus.cc:2229] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:38:05.309819 7885 raft_consensus.cc:2258] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:05.309880 7883 raft_consensus.cc:2258] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:05.310288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 7273
I20260619 18:38:05.310359 7884 ts_tablet_manager.cc:1905] T 570543214a1d44ada34daa1a67ece96c P 5effeb3931dd4e3189b71d2b8e2f40b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:05.310412 7885 ts_tablet_manager.cc:1905] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:05.311863 7883 ts_tablet_manager.cc:1905] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:05.312893 7885 ts_tablet_manager.cc:1918] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:38:05.312969 7885 log.cc:1196] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/570543214a1d44ada34daa1a67ece96c
I20260619 18:38:05.313243 7885 ts_tablet_manager.cc:1939] T 570543214a1d44ada34daa1a67ece96c P 4a3e74f28b6c4b28996a66c22fd0bba4: Deleting consensus metadata
I20260619 18:38:05.314150 7883 ts_tablet_manager.cc:1918] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:38:05.314163 7299 catalog_manager.cc:4841] TS 4a3e74f28b6c4b28996a66c22fd0bba4 (127.0.98.2:35411): tablet 570543214a1d44ada34daa1a67ece96c (table test-table [id=3b4c495fc1974204b7468c6ce1afbaef]) successfully deleted
I20260619 18:38:05.314224 7883 log.cc:1196] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/570543214a1d44ada34daa1a67ece96c
I20260619 18:38:05.314457 7883 ts_tablet_manager.cc:1939] T 570543214a1d44ada34daa1a67ece96c P 97a8dd9ef60d490f8ef3a7f44c47c099: Deleting consensus metadata
I20260619 18:38:05.314998 7296 catalog_manager.cc:4841] TS 97a8dd9ef60d490f8ef3a7f44c47c099 (127.0.98.3:33855): tablet 570543214a1d44ada34daa1a67ece96c (table test-table [id=3b4c495fc1974204b7468c6ce1afbaef]) successfully deleted
I20260619 18:38:05.316864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 7499
I20260619 18:38:05.317102 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:05.316844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42019 (local address 127.0.0.1:44494)
0619 18:38:05.316968 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:05.317024 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 18:38:05.317829 7299 catalog_manager.cc:4584] TS 5effeb3931dd4e3189b71d2b8e2f40b5 (127.0.98.1:42019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 570543214a1d44ada34daa1a67ece96c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42019: connect: Connection refused (error 111)
I20260619 18:38:05.323449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 7635
I20260619 18:38:05.328258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 6985
I20260619 18:38:05.332981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 7099
2026-06-19T18:38:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5024 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:38:05 dist-test-slave-rxv6 krb5kdc[7888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:38:05 dist-test-slave-rxv6 krb5kdc[7888](info): set up 2 sockets
Jun 19 18:38:05 dist-test-slave-rxv6 krb5kdc[7888](info): commencing operation
krb5kdc: starting...
W20260619 18:38:06.290930 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:38:06 dist-test-slave-rxv6 krb5kdc[7888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:38:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:38:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 18:38:06.425843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46527
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:06.555078 7901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:06.555392 7901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:06.555433 7901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:06.559456 7901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:38:06.559521 7901 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 18:38:06.559537 7901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:06.559553 7901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:06.559568 7901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:38:06.559582 7901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:38:06.564980 7901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46527
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:06.566156 7901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:06.567235 7901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:06.576336 7909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:06.576310 7907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:06.576517 7906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:06.576673 7901 server_base.cc:1034] running on GCE node
I20260619 18:38:06.577296 7901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:06.578537 7901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:06.579720 7901 hybrid_clock.cc:648] HybridClock initialized: now 1781894286579694 us; error 53 us; skew 500 ppm
Jun 19 18:38:06 dist-test-slave-rxv6 krb5kdc[7888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894286, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:06.586733 7901 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 18:38:06.587774 7901 webserver.cc:460] Webserver started at http://127.0.98.62:44563/ using document root <none> and password file <none>
I20260619 18:38:06.588411 7901 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:06.588485 7901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:06.588726 7901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:06.591078 7901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "f989fe2fe0e844c1b6f75886d9b3e1c4"
format_stamp: "Formatted at 2026-06-19 18:38:06 on dist-test-slave-rxv6"
server_key: "60c2f77058744f645c5cda00b40a55a7"
server_key_iv: "d5a3ebf535c2dc558646b6a8e9c92c91"
server_key_version: "encryptionkey@0"
I20260619 18:38:06.591650 7901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "f989fe2fe0e844c1b6f75886d9b3e1c4"
format_stamp: "Formatted at 2026-06-19 18:38:06 on dist-test-slave-rxv6"
server_key: "60c2f77058744f645c5cda00b40a55a7"
server_key_iv: "d5a3ebf535c2dc558646b6a8e9c92c91"
server_key_version: "encryptionkey@0"
I20260619 18:38:06.595628 7901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:38:06.598301 7916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:06.599720 7901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:38:06.599849 7901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "f989fe2fe0e844c1b6f75886d9b3e1c4"
format_stamp: "Formatted at 2026-06-19 18:38:06 on dist-test-slave-rxv6"
server_key: "60c2f77058744f645c5cda00b40a55a7"
server_key_iv: "d5a3ebf535c2dc558646b6a8e9c92c91"
server_key_version: "encryptionkey@0"
I20260619 18:38:06.599977 7901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:06.624038 7901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:06.627342 7901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:06.627529 7901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:06.635233 7901 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46527
I20260619 18:38:06.635263 7978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46527 every 8 connection(s)
I20260619 18:38:06.636200 7901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:38:06.639264 7979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:06.644301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 7901
I20260619 18:38:06.644452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:38:06.644737 392 external_mini_cluster.cc:1428] Setting key 4ae8dd5a725e654e7676f02a9e207f8d
I20260619 18:38:06.645309 7979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4: Bootstrap starting.
I20260619 18:38:06.647943 7979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:06.648690 7979 log.cc:824] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:06.650599 7979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4: No bootstrap required, opened a new log
Jun 19 18:38:06 dist-test-slave-rxv6 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 18:38:06.654111 7979 raft_consensus.cc:348] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f989fe2fe0e844c1b6f75886d9b3e1c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46527 } }
I20260619 18:38:06.654369 7979 raft_consensus.cc:374] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:06.654428 7979 raft_consensus.cc:729] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f989fe2fe0e844c1b6f75886d9b3e1c4, State: Initialized, Role: FOLLOWER
I20260619 18:38:06.654896 7979 consensus_queue.cc:260] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f989fe2fe0e844c1b6f75886d9b3e1c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46527 } }
I20260619 18:38:06.655031 7979 raft_consensus.cc:388] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:38:06.655098 7979 raft_consensus.cc:482] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:38:06.655195 7979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:06.656075 7979 raft_consensus.cc:504] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f989fe2fe0e844c1b6f75886d9b3e1c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46527 } }
I20260619 18:38:06.656405 7979 leader_election.cc:302] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f989fe2fe0e844c1b6f75886d9b3e1c4; no voters:
I20260619 18:38:06.656742 7979 leader_election.cc:288] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:38:06.656836 7984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:38:06.657059 7984 raft_consensus.cc:686] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [term 1 LEADER]: Becoming Leader. State: Replica: f989fe2fe0e844c1b6f75886d9b3e1c4, State: Running, Role: LEADER
I20260619 18:38:06.657392 7984 consensus_queue.cc:237] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f989fe2fe0e844c1b6f75886d9b3e1c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46527 } }
I20260619 18:38:06.658195 7979 sys_catalog.cc:564] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:38:06.659149 7985 sys_catalog.cc:455] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f989fe2fe0e844c1b6f75886d9b3e1c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f989fe2fe0e844c1b6f75886d9b3e1c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46527 } } }
I20260619 18:38:06.659149 7986 sys_catalog.cc:455] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f989fe2fe0e844c1b6f75886d9b3e1c4. Latest consensus state: current_term: 1 leader_uuid: "f989fe2fe0e844c1b6f75886d9b3e1c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f989fe2fe0e844c1b6f75886d9b3e1c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46527 } } }
I20260619 18:38:06.659293 7986 sys_catalog.cc:458] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:06.659301 7985 sys_catalog.cc:458] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:06.659567 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:06.646248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.0.98.62:46527)
0619 18:38:06.646705 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:06.646721 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:06.646760 (+ 39us) server_negotiation.cc:407] Connection header received
0619 18:38:06.647441 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:06.647460 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:06.647872 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:06.648243 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:06.649037 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:06.650041 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:06.650898 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:06.651375 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:06.651525 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:06.653851 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:06.653857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:06.653872 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:06.653899 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:06.655761 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:06.656182 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:06.656189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:06.656195 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:06.656331 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:06.656610 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:06.656614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:06.656615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:06.657175 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:06.658010 (+ 835us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:06.658163 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 18:38:06.658487 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":124,"threads_started":1}
I20260619 18:38:06.659880 7993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:38:06.663326 7993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:38:06.670171 7993 catalog_manager.cc:1347] Generated new cluster ID: 8e9ac1c1d703496d8d30d922fad40a07
I20260619 18:38:06.670257 7993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:38:06.682780 7993 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:38:06.684087 7993 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:38:06.695714 7993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4: Generated new TSK 0
I20260619 18:38:06.696578 7993 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 18:38:06.739056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46527
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:06.869472 8005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:06.869755 8005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:06.869798 8005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:06.873970 8005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:06.874037 8005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:06.874078 8005 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:38:06.874117 8005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:38:06.879568 8005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:06.880630 8005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:06.881682 8005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:06.891113 8010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:06.891135 8005 server_base.cc:1034] running on GCE node
W20260619 18:38:06.891182 8011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:06.891253 8013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:06.891760 8005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:06.892414 8005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:06.893581 8005 hybrid_clock.cc:648] HybridClock initialized: now 1781894286893562 us; error 34 us; skew 500 ppm
Jun 19 18:38:06 dist-test-slave-rxv6 krb5kdc[7888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894286, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:06.896481 8005 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 18:38:06.897343 8005 webserver.cc:460] Webserver started at http://127.0.98.1:44191/ using document root <none> and password file <none>
I20260619 18:38:06.897871 8005 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:06.898013 8005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:06.898224 8005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:06.900225 8005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "0bb7854dc2714e08b1da22afdbae90c8"
format_stamp: "Formatted at 2026-06-19 18:38:06 on dist-test-slave-rxv6"
server_key: "14df38dedad966d77f9b3c4017ba94b6"
server_key_iv: "9db29c1a4317c88d5109e87082766c71"
server_key_version: "encryptionkey@0"
I20260619 18:38:06.900768 8005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0bb7854dc2714e08b1da22afdbae90c8"
format_stamp: "Formatted at 2026-06-19 18:38:06 on dist-test-slave-rxv6"
server_key: "14df38dedad966d77f9b3c4017ba94b6"
server_key_iv: "9db29c1a4317c88d5109e87082766c71"
server_key_version: "encryptionkey@0"
I20260619 18:38:06.904503 8005 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 18:38:06.907102 8020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:06.908428 8005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:38:06.908550 8005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "0bb7854dc2714e08b1da22afdbae90c8"
format_stamp: "Formatted at 2026-06-19 18:38:06 on dist-test-slave-rxv6"
server_key: "14df38dedad966d77f9b3c4017ba94b6"
server_key_iv: "9db29c1a4317c88d5109e87082766c71"
server_key_version: "encryptionkey@0"
I20260619 18:38:06.908664 8005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:06.933768 8005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:06.937188 8005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:06.937373 8005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:06.938987 8005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:06.939038 8005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:06.939092 8005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:06.939121 8005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:06.949121 8005 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40411
I20260619 18:38:06.949193 8135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40411 every 8 connection(s)
Jun 19 18:38:06 dist-test-slave-rxv6 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894286, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 18:38:06.950088 8005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:38:06.953354 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:06.939858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53301 (local address 127.0.98.62:46527)
0619 18:38:06.940021 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:06.940025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:06.941075 (+ 1050us) server_negotiation.cc:407] Connection header received
0619 18:38:06.942174 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:06.942177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:06.942235 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:06.942316 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:06.943940 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:06.944754 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:06.946577 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:06.946812 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:06.946901 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:06.949877 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:06.949881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:06.949883 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:06.949943 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:06.951447 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:06.951916 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:06.951920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:06.951922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:06.952002 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:06.952303 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:06.952307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:06.952309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:06.952498 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:06.952582 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:06.953066 (+ 484us) server_negotiation.cc:299] Negotiation successful
0619 18:38:06.953193 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:38:06.954176 8029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:06.940385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46527 (local address 127.0.98.1:53301)
0619 18:38:06.940907 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:06.940953 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:06.941887 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:06.942541 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:06.942551 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:06.943024 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:06.943758 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:06.943780 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:06.944903 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:06.944907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:06.946439 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:06.946451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:06.946965 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:06.946971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:06.947187 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:06.947934 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:06.947958 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:06.949706 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:06.951555 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:06.951561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:06.951576 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:06.951820 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:06.952082 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:06.952085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:06.952087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:06.952222 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:06.952592 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:06.952599 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:06.952844 (+ 245us) client_negotiation.cc:769] Sending connection context
0619 18:38:06.953010 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 18:38:06.953326 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":137,"threads_started":1}
I20260619 18:38:06.955461 8136 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46527
I20260619 18:38:06.955619 8136 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:06.956159 8136 heartbeater.cc:511] Master 127.0.98.62:46527 requested a full tablet report, sending...
I20260619 18:38:06.956429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 8005
I20260619 18:38:06.956540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:38:06.956849 392 external_mini_cluster.cc:1428] Setting key 3ef512f4f0f34cfd55b1166a3d90be9c
I20260619 18:38:06.957834 7932 ts_manager.cc:194] Registered new tserver with Master: 0bb7854dc2714e08b1da22afdbae90c8 (127.0.98.1:40411)
I20260619 18:38:06.959208 7932 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53301
I20260619 18:38:06.964653 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:06.957534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39515 (local address 127.0.98.62:46527)
0619 18:38:06.957635 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:06.957639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:06.957895 (+ 256us) server_negotiation.cc:407] Connection header received
0619 18:38:06.958077 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:06.958080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:06.958120 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:06.958178 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:06.958797 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:06.959532 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:06.960482 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:06.960699 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:06.960779 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:06.961782 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:06.961785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:06.961788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:06.961811 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:06.963141 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:06.963528 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:06.963532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:06.963534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:06.963618 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:06.963883 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:06.963886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:06.963888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:06.964072 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:06.964156 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:06.964396 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 18:38:06.964499 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:38:06.964680 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:06.957567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46527 (local address 127.0.98.1:39515)
0619 18:38:06.957821 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:06.957835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:06.957984 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:06.958303 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:06.958308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:06.958523 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:06.958644 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:06.958651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:06.959646 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:06.959649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:06.960371 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:06.960379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:06.960791 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:06.960795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:06.960882 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:06.961329 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:06.961342 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:06.961660 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:06.963240 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:06.963243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:06.963246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:06.963443 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:06.963700 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:06.963703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:06.963705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:06.963786 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:06.964157 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:06.964160 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:06.964250 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 18:38:06.964334 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 18:38:06.964488 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 18:38:06.986150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46527
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:07.096906 8145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:07.097182 8145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:07.097225 8145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:07.101594 8145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:07.101670 8145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:07.101712 8145 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:38:07.101747 8145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:38:07.107313 8145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:07.108357 8145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:07.109340 8145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:07.119378 8153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:07.119521 8145 server_base.cc:1034] running on GCE node
W20260619 18:38:07.119378 8150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:07.119426 8151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:07.120199 8145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:07.120918 8145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:07.122085 8145 hybrid_clock.cc:648] HybridClock initialized: now 1781894287122057 us; error 40 us; skew 500 ppm
Jun 19 18:38:07 dist-test-slave-rxv6 krb5kdc[7888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894287, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:07.125193 8145 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 18:38:07.126502 8145 webserver.cc:460] Webserver started at http://127.0.98.2:41099/ using document root <none> and password file <none>
I20260619 18:38:07.127239 8145 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:07.127331 8145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:07.127573 8145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:07.129776 8145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "0686c947a5224e02b2eff73947972aac"
format_stamp: "Formatted at 2026-06-19 18:38:07 on dist-test-slave-rxv6"
server_key: "4ffddad7f83d75f4109fc9c91b60fa78"
server_key_iv: "b9cce1fff1e2c89253425397be74b250"
server_key_version: "encryptionkey@0"
I20260619 18:38:07.130331 8145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0686c947a5224e02b2eff73947972aac"
format_stamp: "Formatted at 2026-06-19 18:38:07 on dist-test-slave-rxv6"
server_key: "4ffddad7f83d75f4109fc9c91b60fa78"
server_key_iv: "b9cce1fff1e2c89253425397be74b250"
server_key_version: "encryptionkey@0"
I20260619 18:38:07.134199 8145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:38:07.136837 8160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:07.138218 8145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:38:07.138325 8145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "0686c947a5224e02b2eff73947972aac"
format_stamp: "Formatted at 2026-06-19 18:38:07 on dist-test-slave-rxv6"
server_key: "4ffddad7f83d75f4109fc9c91b60fa78"
server_key_iv: "b9cce1fff1e2c89253425397be74b250"
server_key_version: "encryptionkey@0"
I20260619 18:38:07.138429 8145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:07.151232 8145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:07.154498 8145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:07.154659 8145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:07.156176 8145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:07.156231 8145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:07.156283 8145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:07.156307 8145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:38:07 dist-test-slave-rxv6 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894287, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 18:38:07.166744 8145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42927
I20260619 18:38:07.166831 8275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42927 every 8 connection(s)
I20260619 18:38:07.167893 8145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:38:07.171077 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.157131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47037 (local address 127.0.98.62:46527)
0619 18:38:07.157248 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:07.157252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:07.158445 (+ 1193us) server_negotiation.cc:407] Connection header received
0619 18:38:07.159816 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:07.159820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:07.159881 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:07.160071 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:07.161358 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.162067 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.163812 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.164040 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.164143 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.166896 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:07.166900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:07.166903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:07.166942 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:07.168803 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:07.169320 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:07.169323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:07.169324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:07.169448 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:07.169833 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:07.169836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:07.169837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:07.170082 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:07.170173 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:07.170807 (+ 634us) server_negotiation.cc:299] Negotiation successful
0619 18:38:07.170940 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 18:38:07.171854 8169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.157704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46527 (local address 127.0.98.2:47037)
0619 18:38:07.158274 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:07.158339 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:07.159575 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:07.160240 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:07.160250 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:07.160703 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:07.161211 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.161225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.162201 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.162205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.163670 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.163684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.164209 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.164213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.164404 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.165059 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:07.165080 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:07.166665 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:07.168961 (+ 2296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:07.168966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:07.168978 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:07.169224 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:07.169560 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:07.169563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:07.169565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:07.169718 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:07.170179 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:07.170185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:07.170489 (+ 304us) client_negotiation.cc:769] Sending connection context
0619 18:38:07.170667 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 18:38:07.170963 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":167,"threads_started":1}
I20260619 18:38:07.172726 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 8145
I20260619 18:38:07.172827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:38:07.173108 392 external_mini_cluster.cc:1428] Setting key 65d7f0fdd2175fde3ab5e3e3314ad052
I20260619 18:38:07.173308 8276 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46527
I20260619 18:38:07.173444 8276 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:07.174047 8276 heartbeater.cc:511] Master 127.0.98.62:46527 requested a full tablet report, sending...
I20260619 18:38:07.175108 7933 ts_manager.cc:194] Registered new tserver with Master: 0686c947a5224e02b2eff73947972aac (127.0.98.2:42927)
I20260619 18:38:07.175909 7933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47037
I20260619 18:38:07.182394 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.175601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53605 (local address 127.0.98.62:46527)
0619 18:38:07.176211 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:07.176215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:07.176230 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:38:07.176266 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:07.176269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:07.176309 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:07.176391 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:07.176830 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.177329 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.178164 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.178373 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.178466 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.179567 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:07.179570 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:07.179572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:07.179593 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:07.180864 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:07.181244 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:07.181248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:07.181250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:07.181320 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:07.181557 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:07.181561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:07.181562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:07.181765 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:07.181849 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:07.182134 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 18:38:07.182237 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540}
I20260619 18:38:07.182404 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.175527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46527 (local address 127.0.98.2:53605)
0619 18:38:07.175826 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:07.175841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:07.175928 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:07.176410 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:07.176412 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:07.176616 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:07.176721 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.176727 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.177446 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.177450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.178061 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.178070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.178469 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.178472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.178557 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.179052 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:07.179063 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:07.179443 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:07.180970 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:07.180973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:07.180976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:07.181161 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:07.181398 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:07.181401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:07.181403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:07.181483 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:07.181854 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:07.181857 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:07.181979 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 18:38:07.182073 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 18:38:07.182222 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 18:38:07.201277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46527
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:07.311219 8285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:07.311499 8285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:07.311559 8285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:07.315681 8285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:07.315743 8285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:07.315799 8285 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 18:38:07.315857 8285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:38:07.321157 8285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:07.322293 8285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:07.323287 8285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:07.333019 8293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:07.333091 8291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:07.333058 8290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:07.333140 8285 server_base.cc:1034] running on GCE node
I20260619 18:38:07.333657 8285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:07.334357 8285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:07.335529 8285 hybrid_clock.cc:648] HybridClock initialized: now 1781894287335496 us; error 46 us; skew 500 ppm
Jun 19 18:38:07 dist-test-slave-rxv6 krb5kdc[7888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894287, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:07.338317 8285 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 18:38:07.339165 8285 webserver.cc:460] Webserver started at http://127.0.98.3:39849/ using document root <none> and password file <none>
I20260619 18:38:07.339681 8285 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:07.339741 8285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:07.339920 8285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:07.342056 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "7e2d75795ef64a8e80cef4598b29bf96"
format_stamp: "Formatted at 2026-06-19 18:38:07 on dist-test-slave-rxv6"
server_key: "ffd3a1729fb2e2fb0b6b903e739e230b"
server_key_iv: "fe3bb9970acad4aa915bd683b2a27517"
server_key_version: "encryptionkey@0"
I20260619 18:38:07.342681 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7e2d75795ef64a8e80cef4598b29bf96"
format_stamp: "Formatted at 2026-06-19 18:38:07 on dist-test-slave-rxv6"
server_key: "ffd3a1729fb2e2fb0b6b903e739e230b"
server_key_iv: "fe3bb9970acad4aa915bd683b2a27517"
server_key_version: "encryptionkey@0"
I20260619 18:38:07.346602 8285 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260619 18:38:07.349059 8300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:07.350348 8285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260619 18:38:07.350468 8285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "7e2d75795ef64a8e80cef4598b29bf96"
format_stamp: "Formatted at 2026-06-19 18:38:07 on dist-test-slave-rxv6"
server_key: "ffd3a1729fb2e2fb0b6b903e739e230b"
server_key_iv: "fe3bb9970acad4aa915bd683b2a27517"
server_key_version: "encryptionkey@0"
I20260619 18:38:07.350594 8285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:07.372790 8285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:07.376070 8285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:07.376259 8285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:07.377827 8285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:07.377880 8285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:07.377959 8285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:07.377990 8285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 18:38:07 dist-test-slave-rxv6 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894287, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 18:38:07.388131 8285 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44115
I20260619 18:38:07.388165 8415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44115 every 8 connection(s)
I20260619 18:38:07.389048 8285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:38:07.390924 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.378512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40837 (local address 127.0.98.62:46527)
0619 18:38:07.378652 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:07.378657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:07.379536 (+ 879us) server_negotiation.cc:407] Connection header received
0619 18:38:07.380817 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:07.380821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:07.380877 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:07.381022 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:07.382196 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.382989 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.384147 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.384473 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.384584 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.387249 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:07.387252 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:07.387254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:07.387277 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:07.388795 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:07.389339 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:07.389343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:07.389344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:07.389426 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:07.389830 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:07.389833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:07.389835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:07.390098 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:07.390180 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:07.390675 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 18:38:07.390796 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:38:07.391626 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.378882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46527 (local address 127.0.98.3:40837)
0619 18:38:07.379363 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:07.379421 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:07.380580 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:07.381162 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:07.381171 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:07.381509 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:07.382040 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.382056 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.383126 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.383129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.384051 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.384060 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.384633 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.384638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.384814 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.385339 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:07.385354 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:07.387076 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:07.388906 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:07.388911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:07.388925 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:07.389245 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:07.389544 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:07.389547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:07.389549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:07.389720 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:07.390192 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:07.390197 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:07.390410 (+ 213us) client_negotiation.cc:769] Sending connection context
0619 18:38:07.390598 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 18:38:07.390838 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20260619 18:38:07.392657 8416 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46527
I20260619 18:38:07.392781 8416 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:07.393230 8416 heartbeater.cc:511] Master 127.0.98.62:46527 requested a full tablet report, sending...
I20260619 18:38:07.394179 7932 ts_manager.cc:194] Registered new tserver with Master: 7e2d75795ef64a8e80cef4598b29bf96 (127.0.98.3:44115)
I20260619 18:38:07.394762 7932 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40837
I20260619 18:38:07.397853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 8285
I20260619 18:38:07.397979 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:38:07.398312 392 external_mini_cluster.cc:1428] Setting key d5f98b58b598c8d12141ba1459b40921
I20260619 18:38:07.400285 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:38:07.401157 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.394626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46527 (local address 127.0.98.3:51927)
0619 18:38:07.394936 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:07.394947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:07.395039 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:07.395235 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:07.395238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:07.395408 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:07.395498 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.395503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.396189 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.396193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.396740 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.396748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.397146 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.397150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.397236 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.397665 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:07.397678 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:07.398032 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:07.399849 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:07.399852 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:07.399854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:07.400028 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:07.400256 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:07.400259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:07.400261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:07.400335 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:07.400680 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:07.400683 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:07.400767 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 18:38:07.400849 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 18:38:07.400985 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":82,"threads_started":1}
I20260619 18:38:07.401150 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.394700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51927 (local address 127.0.98.62:46527)
0619 18:38:07.394810 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:07.394814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:07.395015 (+ 201us) server_negotiation.cc:407] Connection header received
0619 18:38:07.395108 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:07.395111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:07.395150 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:07.395213 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:07.395593 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.396080 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.396838 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.397057 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.397138 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.398144 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:07.398148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:07.398151 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:07.398174 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:07.399757 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:07.400107 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:07.400110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:07.400112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:07.400183 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:07.400434 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:07.400438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:07.400440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:07.400598 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:07.400679 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:07.400906 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 18:38:07.401007 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 18:38:07 dist-test-slave-rxv6 krb5kdc[7888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894287, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 18:38:07 dist-test-slave-rxv6 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:07 dist-test-slave-rxv6 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:38:07.423242 7982 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:07.416845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.0.98.62:46527)
0619 18:38:07.416955 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:07.416959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:07.417012 (+ 53us) server_negotiation.cc:407] Connection header received
0619 18:38:07.417053 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:07.417055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:07.417094 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:07.417166 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:07.417901 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.418496 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.419427 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.419662 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.419744 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.423086 (+ 3342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58028: BlockingRecv error: recv got EOF from 127.0.0.1:58028 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 18:38:07 dist-test-slave-rxv6 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894287, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 18:38:07.432312 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.424622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58036 (local address 127.0.98.62:46527)
0619 18:38:07.424723 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:07.424726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:07.424785 (+ 59us) server_negotiation.cc:407] Connection header received
0619 18:38:07.424941 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:07.424944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:07.424985 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:07.425056 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:07.425605 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.426204 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.427043 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.427264 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.427351 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.429611 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:07.429614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:07.429616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:07.429641 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:07.430901 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:07.431342 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:07.431345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:07.431346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:07.431424 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:07.431617 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:07.431619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:07.431620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:07.431783 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:07.431883 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:07.432099 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 18:38:07.432191 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 18:38:07.435184 7932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:38:07.438171 7932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:38:07.452922 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.446701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40411 (local address 127.0.0.1:47842)
0619 18:38:07.447619 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:07.447672 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:07.447816 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:07.448714 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:07.448717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:07.448734 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:07.448846 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.448852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.450261 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.450265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.451377 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.451383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.452311 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.452315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.452459 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.452505 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 18:38:07.452627 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 18:38:07.452758 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":5,"spinlock_wait_cycles":7680,"thread_start_us":103,"threads_started":1}
I20260619 18:38:07.452937 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.446872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.0.98.1:40411)
0619 18:38:07.448309 (+ 1437us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:07.448325 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:07.448348 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:38:07.448439 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:07.448445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:07.448597 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:07.448728 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:07.448980 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.450148 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.451484 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.452196 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.452283 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.452413 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:07.452675 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 18:38:07.452788 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1300,"thread_start_us":127,"threads_started":1}
I20260619 18:38:07.453563 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.447003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42927 (local address 127.0.0.1:47772)
0619 18:38:07.448803 (+ 1800us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:07.448816 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:07.448903 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:07.449336 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:07.449339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:07.449356 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:07.449443 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.449448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.450687 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.450690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.451788 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.451798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.453020 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.453200 (+ 180us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.453282 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.453306 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:38:07.453377 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 18:38:07.453441 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1743,"mutex_wait_us":25,"thread_start_us":76,"threads_started":1}
I20260619 18:38:07.453796 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.447369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.0.98.2:42927)
0619 18:38:07.448102 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:07.448110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:07.448914 (+ 804us) server_negotiation.cc:407] Connection header received
0619 18:38:07.448996 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:07.449001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:07.449194 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:07.449343 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:07.449545 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.450566 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.451909 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.452699 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.452788 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.452959 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:07.453439 (+ 480us) server_negotiation.cc:299] Negotiation successful
0619 18:38:07.453537 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":112,"threads_started":1}
I20260619 18:38:07.454286 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.446910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44115 (local address 127.0.0.1:50074)
0619 18:38:07.447633 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:07.447674 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:07.447815 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:07.448570 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:07.448576 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:07.448597 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:07.448735 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.448743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.449892 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.449895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.451362 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.451372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.453866 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.453869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.454010 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.454034 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:38:07.454084 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 18:38:07.454158 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":71,"spinlock_wait_cycles":16000,"thread_start_us":84,"threads_started":1}
I20260619 18:38:07.455181 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.447600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50074 (local address 127.0.98.3:44115)
0619 18:38:07.448084 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:07.448097 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:07.448120 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:38:07.448191 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:07.448197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:07.448417 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:07.448570 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:07.448883 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.449795 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.452468 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.453534 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.454720 (+ 1186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.454891 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:07.454951 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 18:38:07.455043 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":107,"threads_started":1}
I20260619 18:38:07.456266 8070 tablet_service.cc:1467] Processing CreateTablet for tablet 13b820f536964913b0637f11127fa71c (DEFAULT_TABLE table=test-table [id=39114f99de484299815eba8a29145ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:07.456323 8192 tablet_service.cc:1467] Processing CreateTablet for tablet 13b820f536964913b0637f11127fa71c (DEFAULT_TABLE table=test-table [id=39114f99de484299815eba8a29145ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:07.457195 8070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b820f536964913b0637f11127fa71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:07.457194 8192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b820f536964913b0637f11127fa71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:07.459863 8350 tablet_service.cc:1467] Processing CreateTablet for tablet 13b820f536964913b0637f11127fa71c (DEFAULT_TABLE table=test-table [id=39114f99de484299815eba8a29145ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:07.460727 8350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b820f536964913b0637f11127fa71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:07.466303 8444 tablet_bootstrap.cc:492] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac: Bootstrap starting.
I20260619 18:38:07.467692 8445 tablet_bootstrap.cc:492] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96: Bootstrap starting.
I20260619 18:38:07.468750 8444 tablet_bootstrap.cc:654] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:07.469650 8444 log.cc:824] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:07.469868 8446 tablet_bootstrap.cc:492] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8: Bootstrap starting.
I20260619 18:38:07.470170 8445 tablet_bootstrap.cc:654] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:07.471009 8445 log.cc:824] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:07.471689 8446 tablet_bootstrap.cc:654] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:07.471778 8444 tablet_bootstrap.cc:492] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac: No bootstrap required, opened a new log
I20260619 18:38:07.472002 8444 ts_tablet_manager.cc:1397] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260619 18:38:07.472689 8446 log.cc:824] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:07.473094 8445 tablet_bootstrap.cc:492] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96: No bootstrap required, opened a new log
I20260619 18:38:07.473269 8445 ts_tablet_manager.cc:1397] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:38:07.474257 8446 tablet_bootstrap.cc:492] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8: No bootstrap required, opened a new log
I20260619 18:38:07.474411 8446 ts_tablet_manager.cc:1397] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:38:07.475916 8444 raft_consensus.cc:348] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } }
I20260619 18:38:07.476198 8444 raft_consensus.cc:374] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:07.476251 8444 raft_consensus.cc:729] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0686c947a5224e02b2eff73947972aac, State: Initialized, Role: FOLLOWER
I20260619 18:38:07.476730 8444 consensus_queue.cc:260] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } }
I20260619 18:38:07.477063 8445 raft_consensus.cc:348] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } }
I20260619 18:38:07.477301 8445 raft_consensus.cc:374] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:07.477361 8445 raft_consensus.cc:729] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e2d75795ef64a8e80cef4598b29bf96, State: Initialized, Role: FOLLOWER
I20260619 18:38:07.477725 8444 ts_tablet_manager.cc:1428] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260619 18:38:07.477766 8445 consensus_queue.cc:260] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } }
I20260619 18:38:07.477744 8446 raft_consensus.cc:348] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } }
I20260619 18:38:07.477982 8446 raft_consensus.cc:374] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:07.478035 8446 raft_consensus.cc:729] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb7854dc2714e08b1da22afdbae90c8, State: Initialized, Role: FOLLOWER
I20260619 18:38:07.478097 8276 heartbeater.cc:503] Master 127.0.98.62:46527 was elected leader, sending a full tablet report...
I20260619 18:38:07.478372 8446 consensus_queue.cc:260] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } }
I20260619 18:38:07.478693 8445 ts_tablet_manager.cc:1428] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260619 18:38:07.479019 8416 heartbeater.cc:503] Master 127.0.98.62:46527 was elected leader, sending a full tablet report...
I20260619 18:38:07.479102 8136 heartbeater.cc:503] Master 127.0.98.62:46527 was elected leader, sending a full tablet report...
I20260619 18:38:07.479171 8446 ts_tablet_manager.cc:1428] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260619 18:38:07.640067 8417 tablet.cc:2369] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:38:07.669129 8277 tablet.cc:2369] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:38:07.698738 8450 raft_consensus.cc:482] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:38:07.698971 8450 raft_consensus.cc:504] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } }
I20260619 18:38:07.700065 8450 leader_election.cc:288] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bb7854dc2714e08b1da22afdbae90c8 (127.0.98.1:40411), 7e2d75795ef64a8e80cef4598b29bf96 (127.0.98.3:44115)
W20260619 18:38:07.701818 8137 tablet.cc:2369] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:38:07.703907 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.700405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48547 (local address 127.0.98.1:40411)
0619 18:38:07.700555 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:07.700560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:07.700720 (+ 160us) server_negotiation.cc:407] Connection header received
0619 18:38:07.700873 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:07.700876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:07.700925 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:07.700981 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:07.701283 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.701818 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.702930 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.703320 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.703417 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.703492 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:07.703725 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 18:38:07.703797 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:38:07.703926 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.700317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40411 (local address 127.0.98.2:48547)
0619 18:38:07.700609 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:07.700624 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:07.700752 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:07.701070 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:07.701073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:07.701090 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:07.701186 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.701191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.701938 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.701942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.702816 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.702825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.703492 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.703495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.703602 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.703628 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:38:07.703691 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 18:38:07.703767 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260619 18:38:07.704069 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.700438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42659 (local address 127.0.98.3:44115)
0619 18:38:07.700659 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:07.700663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:07.700773 (+ 110us) server_negotiation.cc:407] Connection header received
0619 18:38:07.700894 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:07.700896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:07.700935 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:07.701002 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:07.701258 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.701854 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.703019 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.703496 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.703610 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.703669 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:07.703832 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 18:38:07.703904 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260619 18:38:07.704209 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.700317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44115 (local address 127.0.98.2:42659)
0619 18:38:07.700680 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:07.700696 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:07.700801 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:07.701025 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:07.701028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:07.701043 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:07.701161 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.701166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.702026 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.702029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.702890 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.702900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.703608 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.703611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.703716 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.703737 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:38:07.703799 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 18:38:07.703869 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":170,"thread_start_us":46,"threads_started":1}
I20260619 18:38:07.704727 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b820f536964913b0637f11127fa71c" candidate_uuid: "0686c947a5224e02b2eff73947972aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb7854dc2714e08b1da22afdbae90c8" is_pre_election: true
I20260619 18:38:07.704874 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b820f536964913b0637f11127fa71c" candidate_uuid: "0686c947a5224e02b2eff73947972aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e2d75795ef64a8e80cef4598b29bf96" is_pre_election: true
I20260619 18:38:07.705025 8090 raft_consensus.cc:2454] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0686c947a5224e02b2eff73947972aac in term 0.
I20260619 18:38:07.705186 8370 raft_consensus.cc:2454] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0686c947a5224e02b2eff73947972aac in term 0.
I20260619 18:38:07.705466 8161 leader_election.cc:302] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0686c947a5224e02b2eff73947972aac, 0bb7854dc2714e08b1da22afdbae90c8; no voters:
I20260619 18:38:07.705763 8450 raft_consensus.cc:2781] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:38:07.705844 8450 raft_consensus.cc:482] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:38:07.705892 8450 raft_consensus.cc:3037] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:07.706920 8450 raft_consensus.cc:504] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } }
I20260619 18:38:07.707245 8450 leader_election.cc:288] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [CANDIDATE]: Term 1 election: Requested vote from peers 0bb7854dc2714e08b1da22afdbae90c8 (127.0.98.1:40411), 7e2d75795ef64a8e80cef4598b29bf96 (127.0.98.3:44115)
I20260619 18:38:07.707547 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b820f536964913b0637f11127fa71c" candidate_uuid: "0686c947a5224e02b2eff73947972aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb7854dc2714e08b1da22afdbae90c8"
I20260619 18:38:07.707561 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13b820f536964913b0637f11127fa71c" candidate_uuid: "0686c947a5224e02b2eff73947972aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e2d75795ef64a8e80cef4598b29bf96"
I20260619 18:38:07.707655 8370 raft_consensus.cc:3037] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:07.707664 8090 raft_consensus.cc:3037] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:07.708684 8090 raft_consensus.cc:2454] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0686c947a5224e02b2eff73947972aac in term 1.
I20260619 18:38:07.708853 8370 raft_consensus.cc:2454] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0686c947a5224e02b2eff73947972aac in term 1.
I20260619 18:38:07.708963 8161 leader_election.cc:302] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0686c947a5224e02b2eff73947972aac, 0bb7854dc2714e08b1da22afdbae90c8; no voters:
I20260619 18:38:07.709128 8450 raft_consensus.cc:2781] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:38:07.709288 8450 raft_consensus.cc:686] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [term 1 LEADER]: Becoming Leader. State: Replica: 0686c947a5224e02b2eff73947972aac, State: Running, Role: LEADER
I20260619 18:38:07.709599 8450 consensus_queue.cc:237] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } }
I20260619 18:38:07.712882 7932 catalog_manager.cc:5495] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac reported cstate change: term changed from 0 to 1, leader changed from <none> to 0686c947a5224e02b2eff73947972aac (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0686c947a5224e02b2eff73947972aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } health_report { overall_health: HEALTHY } } }
I20260619 18:38:07.744050 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.740479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.0.98.2:42927)
0619 18:38:07.740605 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:07.740609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:07.740631 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:38:07.740757 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:07.740759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:07.740817 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:07.740909 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:38:07.741132 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.741669 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.742607 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.742855 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.742954 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.743213 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:38:07.743650 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:38:07.743756 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:07.743834 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 18:38:07.743893 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:38:07.749815 8090 raft_consensus.cc:1261] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 FOLLOWER]: Refusing update from remote peer 0686c947a5224e02b2eff73947972aac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:38:07.749815 8370 raft_consensus.cc:1261] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Refusing update from remote peer 0686c947a5224e02b2eff73947972aac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:38:07.750520 8450 consensus_queue.cc:1035] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:38:07.750705 8455 consensus_queue.cc:1035] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:38:07.757282 8458 mvcc.cc:205] Tried to move back new op lower bound from 7298639002616025088 to 7298639002461241344. Current Snapshot: MvccSnapshot[applied={T|T < 7298639002616025088}]
I20260619 18:38:07.758526 8460 mvcc.cc:205] Tried to move back new op lower bound from 7298639002616025088 to 7298639002461241344. Current Snapshot: MvccSnapshot[applied={T|T < 7298639002616025088}]
I20260619 18:38:07.759374 8459 mvcc.cc:205] Tried to move back new op lower bound from 7298639002616025088 to 7298639002461241344. Current Snapshot: MvccSnapshot[applied={T|T < 7298639002616025088}]
I20260619 18:38:07.765061 7933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:38:07.765232 7933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:38:07.767560 7933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4: Sending DeleteTablet for 3 replicas of tablet 13b820f536964913b0637f11127fa71c
I20260619 18:38:07.768220 8350 tablet_service.cc:1514] Processing DeleteTablet for tablet 13b820f536964913b0637f11127fa71c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50074
I20260619 18:38:07.768219 8070 tablet_service.cc:1514] Processing DeleteTablet for tablet 13b820f536964913b0637f11127fa71c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47842
I20260619 18:38:07.768697 8468 tablet_replica.cc:330] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8: stopping tablet replica
I20260619 18:38:07.768922 8469 tablet_replica.cc:330] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96: stopping tablet replica
I20260619 18:38:07.768935 8192 tablet_service.cc:1514] Processing DeleteTablet for tablet 13b820f536964913b0637f11127fa71c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47772
I20260619 18:38:07.769287 8469 raft_consensus.cc:2229] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:38:07.769266 7933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 18:38:07.769366 8470 tablet_replica.cc:330] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac: stopping tablet replica
I20260619 18:38:07.769608 8469 raft_consensus.cc:2258] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:07.769609 8470 raft_consensus.cc:2229] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [term 1 LEADER]: Raft consensus shutting down.
W20260619 18:38:07.769640 7933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:38:07.770009 8470 raft_consensus.cc:2258] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:07.770047 8468 raft_consensus.cc:2229] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:38:07.770298 8468 raft_consensus.cc:2258] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:07.772038 8468 ts_tablet_manager.cc:1905] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:07.773175 8470 ts_tablet_manager.cc:1905] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:07.774389 8469 ts_tablet_manager.cc:1905] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:07.774607 8070 tablet_service.cc:1467] Processing CreateTablet for tablet 8792ceee5f454cb7a136dc5202ffb38e (DEFAULT_TABLE table=test-table [id=98ab403e81094f999cbece1bf266ef09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:07.774832 8070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8792ceee5f454cb7a136dc5202ffb38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:07.776544 8446 tablet_bootstrap.cc:492] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8: Bootstrap starting.
I20260619 18:38:07.776876 8469 ts_tablet_manager.cc:1918] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:38:07.776970 8469 log.cc:1196] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/13b820f536964913b0637f11127fa71c
I20260619 18:38:07.777238 8469 ts_tablet_manager.cc:1939] T 13b820f536964913b0637f11127fa71c P 7e2d75795ef64a8e80cef4598b29bf96: Deleting consensus metadata
I20260619 18:38:07.777477 8446 tablet_bootstrap.cc:654] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:07.777558 8192 tablet_service.cc:1467] Processing CreateTablet for tablet 8792ceee5f454cb7a136dc5202ffb38e (DEFAULT_TABLE table=test-table [id=98ab403e81094f999cbece1bf266ef09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:07.777786 8192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8792ceee5f454cb7a136dc5202ffb38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:07.778482 7920 catalog_manager.cc:4841] TS 7e2d75795ef64a8e80cef4598b29bf96 (127.0.98.3:44115): tablet 13b820f536964913b0637f11127fa71c (table test-table [id=39114f99de484299815eba8a29145ec9]) successfully deleted
I20260619 18:38:07.779604 8468 ts_tablet_manager.cc:1918] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:38:07.779687 8468 log.cc:1196] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/wal/wals/13b820f536964913b0637f11127fa71c
I20260619 18:38:07.779713 8444 tablet_bootstrap.cc:492] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac: Bootstrap starting.
I20260619 18:38:07.779974 8350 tablet_service.cc:1467] Processing CreateTablet for tablet 8792ceee5f454cb7a136dc5202ffb38e (DEFAULT_TABLE table=test-table [id=98ab403e81094f999cbece1bf266ef09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:07.780192 8350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8792ceee5f454cb7a136dc5202ffb38e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:07.780364 8468 ts_tablet_manager.cc:1939] T 13b820f536964913b0637f11127fa71c P 0bb7854dc2714e08b1da22afdbae90c8: Deleting consensus metadata
I20260619 18:38:07.780576 8444 tablet_bootstrap.cc:654] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:07.781433 7917 catalog_manager.cc:4841] TS 0bb7854dc2714e08b1da22afdbae90c8 (127.0.98.1:40411): tablet 13b820f536964913b0637f11127fa71c (table test-table [id=39114f99de484299815eba8a29145ec9]) successfully deleted
I20260619 18:38:07.782222 8445 tablet_bootstrap.cc:492] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96: Bootstrap starting.
I20260619 18:38:07.783121 8445 tablet_bootstrap.cc:654] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:07.783574 8446 tablet_bootstrap.cc:492] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8: No bootstrap required, opened a new log
I20260619 18:38:07.783648 8446 ts_tablet_manager.cc:1397] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260619 18:38:07.784051 8446 raft_consensus.cc:348] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:07.784176 8446 raft_consensus.cc:374] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:07.784216 8446 raft_consensus.cc:729] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb7854dc2714e08b1da22afdbae90c8, State: Initialized, Role: FOLLOWER
I20260619 18:38:07.784340 8446 consensus_queue.cc:260] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:07.784359 8470 ts_tablet_manager.cc:1918] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:38:07.784442 8470 log.cc:1196] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/13b820f536964913b0637f11127fa71c
I20260619 18:38:07.784574 8446 ts_tablet_manager.cc:1428] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 18:38:07.784715 8470 ts_tablet_manager.cc:1939] T 13b820f536964913b0637f11127fa71c P 0686c947a5224e02b2eff73947972aac: Deleting consensus metadata
I20260619 18:38:07.785840 7919 catalog_manager.cc:4841] TS 0686c947a5224e02b2eff73947972aac (127.0.98.2:42927): tablet 13b820f536964913b0637f11127fa71c (table test-table [id=39114f99de484299815eba8a29145ec9]) successfully deleted
I20260619 18:38:07.786597 8444 tablet_bootstrap.cc:492] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac: No bootstrap required, opened a new log
I20260619 18:38:07.786612 8445 tablet_bootstrap.cc:492] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96: No bootstrap required, opened a new log
I20260619 18:38:07.786685 8445 ts_tablet_manager.cc:1397] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 18:38:07.786684 8444 ts_tablet_manager.cc:1397] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260619 18:38:07.787093 8445 raft_consensus.cc:348] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:07.787093 8444 raft_consensus.cc:348] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:07.787200 8444 raft_consensus.cc:374] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:07.787200 8445 raft_consensus.cc:374] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:07.787240 8444 raft_consensus.cc:729] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0686c947a5224e02b2eff73947972aac, State: Initialized, Role: FOLLOWER
I20260619 18:38:07.787240 8445 raft_consensus.cc:729] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e2d75795ef64a8e80cef4598b29bf96, State: Initialized, Role: FOLLOWER
I20260619 18:38:07.787353 8444 consensus_queue.cc:260] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:07.787355 8445 consensus_queue.cc:260] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:07.787591 8444 ts_tablet_manager.cc:1428] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:38:07.787604 8445 ts_tablet_manager.cc:1428] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:38:07.811751 8452 raft_consensus.cc:482] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:38:07.811892 8452 raft_consensus.cc:504] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:07.813038 8452 leader_election.cc:288] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0686c947a5224e02b2eff73947972aac (127.0.98.2:42927), 7e2d75795ef64a8e80cef4598b29bf96 (127.0.98.3:44115)
I20260619 18:38:07.817139 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.813232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42927 (local address 127.0.98.1:43047)
0619 18:38:07.813538 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:07.813552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:07.813701 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:07.814111 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:07.814114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:07.814133 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:07.814257 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.814263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.815220 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.815223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.816081 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.816090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.816709 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.816712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.816811 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.816835 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:38:07.816907 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 18:38:07.816973 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":79,"threads_started":1}
I20260619 18:38:07.817150 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.813304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43047 (local address 127.0.98.2:42927)
0619 18:38:07.813420 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:07.813425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:07.813621 (+ 196us) server_negotiation.cc:407] Connection header received
0619 18:38:07.813824 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:07.813827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:07.813877 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:07.813960 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:07.814362 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.815107 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.816198 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.816580 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.816671 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.816735 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:07.816940 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 18:38:07.817025 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:38:07.817894 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.813679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43757 (local address 127.0.98.3:44115)
0619 18:38:07.813800 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:07.813804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:07.814288 (+ 484us) server_negotiation.cc:407] Connection header received
0619 18:38:07.814451 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:07.814454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:07.814493 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:07.814558 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:07.814811 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.815329 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.816794 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.817346 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.817438 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.817493 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:07.817713 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 18:38:07.817784 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:38:07.817934 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.813371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44115 (local address 127.0.98.1:43757)
0619 18:38:07.814203 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:07.814217 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:07.814347 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:07.814584 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:07.814587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:07.814602 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:07.814715 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.814719 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.815438 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.815442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.816643 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.816654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.817455 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.817459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.817561 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.817585 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:38:07.817691 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 18:38:07.817753 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":89,"threads_started":1}
I20260619 18:38:07.818001 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8792ceee5f454cb7a136dc5202ffb38e" candidate_uuid: "0bb7854dc2714e08b1da22afdbae90c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0686c947a5224e02b2eff73947972aac" is_pre_election: true
I20260619 18:38:07.818243 8230 raft_consensus.cc:2454] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bb7854dc2714e08b1da22afdbae90c8 in term 0.
I20260619 18:38:07.818279 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8792ceee5f454cb7a136dc5202ffb38e" candidate_uuid: "0bb7854dc2714e08b1da22afdbae90c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e2d75795ef64a8e80cef4598b29bf96" is_pre_election: true
I20260619 18:38:07.818405 8370 raft_consensus.cc:2454] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bb7854dc2714e08b1da22afdbae90c8 in term 0.
I20260619 18:38:07.818679 8023 leader_election.cc:302] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0686c947a5224e02b2eff73947972aac, 0bb7854dc2714e08b1da22afdbae90c8; no voters:
I20260619 18:38:07.818935 8452 raft_consensus.cc:2781] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:38:07.819020 8452 raft_consensus.cc:482] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:38:07.819067 8452 raft_consensus.cc:3037] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:07.819973 8452 raft_consensus.cc:504] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:07.820320 8452 leader_election.cc:288] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [CANDIDATE]: Term 1 election: Requested vote from peers 0686c947a5224e02b2eff73947972aac (127.0.98.2:42927), 7e2d75795ef64a8e80cef4598b29bf96 (127.0.98.3:44115)
I20260619 18:38:07.820652 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8792ceee5f454cb7a136dc5202ffb38e" candidate_uuid: "0bb7854dc2714e08b1da22afdbae90c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e2d75795ef64a8e80cef4598b29bf96"
I20260619 18:38:07.820645 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8792ceee5f454cb7a136dc5202ffb38e" candidate_uuid: "0bb7854dc2714e08b1da22afdbae90c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0686c947a5224e02b2eff73947972aac"
I20260619 18:38:07.820755 8230 raft_consensus.cc:3037] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:07.820757 8370 raft_consensus.cc:3037] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:07.821584 8370 raft_consensus.cc:2454] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bb7854dc2714e08b1da22afdbae90c8 in term 1.
I20260619 18:38:07.821841 8024 leader_election.cc:302] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb7854dc2714e08b1da22afdbae90c8, 7e2d75795ef64a8e80cef4598b29bf96; no voters:
I20260619 18:38:07.821889 8230 raft_consensus.cc:2454] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bb7854dc2714e08b1da22afdbae90c8 in term 1.
I20260619 18:38:07.822038 8452 raft_consensus.cc:2781] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:38:07.822244 8452 raft_consensus.cc:686] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0bb7854dc2714e08b1da22afdbae90c8, State: Running, Role: LEADER
I20260619 18:38:07.822522 8452 consensus_queue.cc:237] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:07.824710 7933 catalog_manager.cc:5495] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bb7854dc2714e08b1da22afdbae90c8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0bb7854dc2714e08b1da22afdbae90c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } health_report { overall_health: UNKNOWN } } }
Jun 19 18:38:07 dist-test-slave-rxv6 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894286, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 18:38:07.876731 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.867607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46527 (local address 127.0.0.1:58040)
0619 18:38:07.867885 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:07.867899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:07.868001 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:07.868281 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:07.868284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:07.868678 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:07.868775 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.868780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.869765 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.869768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.870405 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:07.870414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.870966 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.870969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:07.871064 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.871626 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:07.871637 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:07.873001 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:07.875084 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:07.875091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:07.875097 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:07.875337 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:07.875632 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:07.875634 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:07.875635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:07.875715 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:07.876215 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:07.876221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:07.876334 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 18:38:07.876437 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 18:38:07.876570 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
I20260619 18:38:07.877009 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:07.867698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58040 (local address 127.0.98.62:46527)
0619 18:38:07.867842 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:07.867847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:07.867976 (+ 129us) server_negotiation.cc:407] Connection header received
0619 18:38:07.868091 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:07.868095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:07.868143 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:07.868241 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:07.868898 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.869642 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.870514 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:07.870847 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:07.870952 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:07.873148 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:07.873152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:07.873155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:07.873180 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:07.874915 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:07.875445 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:07.875449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:07.875451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:07.875546 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:07.875799 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:07.875802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:07.875804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:07.876060 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:07.876168 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:07.876523 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 18:38:07.876651 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:38:07.880788 7933 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58040:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 18:38:07.881618 7933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:38:07.886374 8070 tablet_service.cc:1467] Processing CreateTablet for tablet 1343cedfd3ea49f7b3d4fee3013f323f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b70655be2b194e1cbb7c51230c71df20]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:38:07.886598 8350 tablet_service.cc:1467] Processing CreateTablet for tablet 1343cedfd3ea49f7b3d4fee3013f323f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b70655be2b194e1cbb7c51230c71df20]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:38:07.886803 8070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1343cedfd3ea49f7b3d4fee3013f323f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:07.886682 8192 tablet_service.cc:1467] Processing CreateTablet for tablet 1343cedfd3ea49f7b3d4fee3013f323f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b70655be2b194e1cbb7c51230c71df20]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 18:38:07.886965 8350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1343cedfd3ea49f7b3d4fee3013f323f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:07.886988 8192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1343cedfd3ea49f7b3d4fee3013f323f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:07.889657 8446 tablet_bootstrap.cc:492] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8: Bootstrap starting.
I20260619 18:38:07.890136 8445 tablet_bootstrap.cc:492] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96: Bootstrap starting.
I20260619 18:38:07.890637 8446 tablet_bootstrap.cc:654] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:07.891080 8445 tablet_bootstrap.cc:654] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:07.891485 8444 tablet_bootstrap.cc:492] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac: Bootstrap starting.
I20260619 18:38:07.892366 8444 tablet_bootstrap.cc:654] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:07.892364 8445 tablet_bootstrap.cc:492] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96: No bootstrap required, opened a new log
I20260619 18:38:07.892448 8445 ts_tablet_manager.cc:1397] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 18:38:07.892910 8445 raft_consensus.cc:348] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:07.893026 8445 raft_consensus.cc:374] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:07.893064 8445 raft_consensus.cc:729] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e2d75795ef64a8e80cef4598b29bf96, State: Initialized, Role: FOLLOWER
I20260619 18:38:07.893180 8445 consensus_queue.cc:260] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:07.893242 8446 tablet_bootstrap.cc:492] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8: No bootstrap required, opened a new log
I20260619 18:38:07.893311 8446 ts_tablet_manager.cc:1397] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 18:38:07.893443 8445 ts_tablet_manager.cc:1428] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:38:07.893709 8446 raft_consensus.cc:348] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:07.893689 8451 tablet_replica.cc:439] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } }
I20260619 18:38:07.893812 8446 raft_consensus.cc:374] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:07.893836 8451 tablet_replica.cc:442] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:38:07.893853 8446 raft_consensus.cc:729] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb7854dc2714e08b1da22afdbae90c8, State: Initialized, Role: FOLLOWER
I20260619 18:38:07.893983 8446 consensus_queue.cc:260] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:07.894230 8446 ts_tablet_manager.cc:1428] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:38:07.894212 8452 tablet_replica.cc:439] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } }
I20260619 18:38:07.894322 8452 tablet_replica.cc:442] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:38:07.895073 8444 tablet_bootstrap.cc:492] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac: No bootstrap required, opened a new log
I20260619 18:38:07.895154 8444 ts_tablet_manager.cc:1397] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 18:38:07.895540 8444 raft_consensus.cc:348] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:07.895658 8444 raft_consensus.cc:374] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:07.895689 8444 raft_consensus.cc:729] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0686c947a5224e02b2eff73947972aac, State: Initialized, Role: FOLLOWER
I20260619 18:38:07.895804 8444 consensus_queue.cc:260] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:07.896396 8444 ts_tablet_manager.cc:1428] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:38:07.896790 8463 tablet_replica.cc:439] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } }
I20260619 18:38:07.896916 8463 tablet_replica.cc:442] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:38:08.096715 8463 raft_consensus.cc:482] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:38:08.096880 8463 raft_consensus.cc:504] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.097296 8463 leader_election.cc:288] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bb7854dc2714e08b1da22afdbae90c8 (127.0.98.1:40411), 7e2d75795ef64a8e80cef4598b29bf96 (127.0.98.3:44115)
I20260619 18:38:08.097779 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1343cedfd3ea49f7b3d4fee3013f323f" candidate_uuid: "0686c947a5224e02b2eff73947972aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb7854dc2714e08b1da22afdbae90c8" is_pre_election: true
I20260619 18:38:08.097782 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1343cedfd3ea49f7b3d4fee3013f323f" candidate_uuid: "0686c947a5224e02b2eff73947972aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e2d75795ef64a8e80cef4598b29bf96" is_pre_election: true
I20260619 18:38:08.097945 8090 raft_consensus.cc:2454] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0686c947a5224e02b2eff73947972aac in term 0.
I20260619 18:38:08.097949 8370 raft_consensus.cc:2454] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0686c947a5224e02b2eff73947972aac in term 0.
I20260619 18:38:08.098284 8164 leader_election.cc:302] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0686c947a5224e02b2eff73947972aac, 7e2d75795ef64a8e80cef4598b29bf96; no voters:
I20260619 18:38:08.098443 8463 raft_consensus.cc:2781] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:38:08.098497 8463 raft_consensus.cc:482] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:38:08.098587 8463 raft_consensus.cc:3037] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:08.099586 8463 raft_consensus.cc:504] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.099903 8463 leader_election.cc:288] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [CANDIDATE]: Term 1 election: Requested vote from peers 0bb7854dc2714e08b1da22afdbae90c8 (127.0.98.1:40411), 7e2d75795ef64a8e80cef4598b29bf96 (127.0.98.3:44115)
I20260619 18:38:08.100245 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1343cedfd3ea49f7b3d4fee3013f323f" candidate_uuid: "0686c947a5224e02b2eff73947972aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e2d75795ef64a8e80cef4598b29bf96"
I20260619 18:38:08.100272 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1343cedfd3ea49f7b3d4fee3013f323f" candidate_uuid: "0686c947a5224e02b2eff73947972aac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb7854dc2714e08b1da22afdbae90c8"
I20260619 18:38:08.100363 8090 raft_consensus.cc:3037] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:08.100373 8370 raft_consensus.cc:3037] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:08.101193 8090 raft_consensus.cc:2454] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0686c947a5224e02b2eff73947972aac in term 1.
I20260619 18:38:08.101249 8370 raft_consensus.cc:2454] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0686c947a5224e02b2eff73947972aac in term 1.
I20260619 18:38:08.101464 8161 leader_election.cc:302] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0686c947a5224e02b2eff73947972aac, 0bb7854dc2714e08b1da22afdbae90c8; no voters:
I20260619 18:38:08.101666 8463 raft_consensus.cc:2781] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:38:08.101742 8463 raft_consensus.cc:686] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [term 1 LEADER]: Becoming Leader. State: Replica: 0686c947a5224e02b2eff73947972aac, State: Running, Role: LEADER
I20260619 18:38:08.101874 8463 consensus_queue.cc:237] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.102561 8450 tablet_replica.cc:439] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac: TxnStatusTablet state changed. Reason: New leader 0686c947a5224e02b2eff73947972aac. Latest consensus state: current_term: 1 leader_uuid: "0686c947a5224e02b2eff73947972aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } }
I20260619 18:38:08.102665 8450 tablet_replica.cc:442] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:38:08.102921 8488 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 18:38:08.103176 8488 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 18:38:08.103634 7933 catalog_manager.cc:5495] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac reported cstate change: term changed from 0 to 1, leader changed from <none> to 0686c947a5224e02b2eff73947972aac (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0686c947a5224e02b2eff73947972aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } health_report { overall_health: UNKNOWN } } }
I20260619 18:38:08.140601 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:08.136866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42927 (local address 127.0.0.1:47784)
0619 18:38:08.137150 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:08.137162 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:08.137258 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:08.137544 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:08.137546 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:08.137555 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:38:08.137642 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:08.137646 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.138652 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.138655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:08.139324 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:08.139334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.139907 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.139910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:08.139981 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:08.140025 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:38:08.140329 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:38:08.140342 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 18:38:08.140398 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 18:38:08.140488 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260619 18:38:08.140606 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:08.136986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.0.98.2:42927)
0619 18:38:08.137117 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:08.137121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:08.137214 (+ 93us) server_negotiation.cc:407] Connection header received
0619 18:38:08.137374 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:08.137377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:08.137430 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:08.137519 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:38:08.137724 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.138527 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.139434 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.139784 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.139886 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:08.140166 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:38:08.140249 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:38:08.140315 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:08.140407 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 18:38:08.140456 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:38:08.143725 8370 raft_consensus.cc:1261] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Refusing update from remote peer 0686c947a5224e02b2eff73947972aac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:38:08.143759 8090 raft_consensus.cc:1261] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 FOLLOWER]: Refusing update from remote peer 0686c947a5224e02b2eff73947972aac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:38:08.144193 8463 consensus_queue.cc:1035] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:38:08.144366 8450 consensus_queue.cc:1035] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:38:08.145707 8452 tablet_replica.cc:439] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8: TxnStatusTablet state changed. Reason: New leader 0686c947a5224e02b2eff73947972aac. Latest consensus state: current_term: 1 leader_uuid: "0686c947a5224e02b2eff73947972aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } }
I20260619 18:38:08.145828 8452 tablet_replica.cc:442] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:38:08.145987 8451 tablet_replica.cc:439] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96: TxnStatusTablet state changed. Reason: New leader 0686c947a5224e02b2eff73947972aac. Latest consensus state: current_term: 1 leader_uuid: "0686c947a5224e02b2eff73947972aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } }
I20260619 18:38:08.146095 8451 tablet_replica.cc:442] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:38:08.147178 8463 tablet_replica.cc:439] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0686c947a5224e02b2eff73947972aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } }
I20260619 18:38:08.147284 8463 tablet_replica.cc:442] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:38:08.147909 8451 tablet_replica.cc:439] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0686c947a5224e02b2eff73947972aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } }
I20260619 18:38:08.148013 8451 tablet_replica.cc:442] T 1343cedfd3ea49f7b3d4fee3013f323f P 7e2d75795ef64a8e80cef4598b29bf96: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:38:08.151000 8463 tablet_replica.cc:439] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0686c947a5224e02b2eff73947972aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } }
I20260619 18:38:08.151103 8463 tablet_replica.cc:442] T 1343cedfd3ea49f7b3d4fee3013f323f P 0686c947a5224e02b2eff73947972aac: This TxnStatusTablet replica's current role is: LEADER
I20260619 18:38:08.152218 8452 tablet_replica.cc:439] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0686c947a5224e02b2eff73947972aac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } } }
I20260619 18:38:08.152328 8452 tablet_replica.cc:442] T 1343cedfd3ea49f7b3d4fee3013f323f P 0bb7854dc2714e08b1da22afdbae90c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 18:38:08.156617 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:08.152778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.0.98.1:40411)
0619 18:38:08.152884 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:08.152888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:08.152909 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:38:08.152978 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:08.152981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:08.153048 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:08.153161 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:38:08.153522 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.154277 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.155217 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.155445 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.155553 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:08.155806 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:38:08.156258 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:38:08.156360 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:08.156427 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 18:38:08.156481 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:38:08.167560 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:08.163823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40375 (local address 127.0.98.2:42927)
0619 18:38:08.163944 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:08.163948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:08.164126 (+ 178us) server_negotiation.cc:407] Connection header received
0619 18:38:08.164226 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:08.164229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:08.164279 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:08.164358 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:38:08.164622 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.165377 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.166363 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.166725 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.166824 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:08.167153 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:38:08.167234 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:38:08.167302 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:08.167374 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 18:38:08.167423 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:38:08.167550 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:08.163705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42927 (local address 127.0.98.1:40375)
0619 18:38:08.164055 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:08.164067 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:08.164148 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:08.164392 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:08.164394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:08.164401 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:38:08.164506 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:08.164510 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.165493 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.165496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:08.166261 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:08.166271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.166906 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.166909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:08.166985 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:08.167028 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:38:08.167307 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:38:08.167310 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 18:38:08.167348 (+ 38us) client_negotiation.cc:240] Negotiation successful
0619 18:38:08.167408 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":123,"threads_started":1}
I20260619 18:38:08.175356 8370 raft_consensus.cc:1261] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Refusing update from remote peer 0bb7854dc2714e08b1da22afdbae90c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:38:08.175571 8230 raft_consensus.cc:1261] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Refusing update from remote peer 0bb7854dc2714e08b1da22afdbae90c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:38:08.175920 8476 consensus_queue.cc:1035] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:38:08.176262 8476 consensus_queue.cc:1035] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:38:08.201551 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:08.194541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58887 (local address 127.0.98.1:40411)
0619 18:38:08.194637 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:08.194641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:08.194689 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:38:08.194731 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:08.194734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:08.194785 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:08.194849 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:38:08.196727 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.197447 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.200412 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.200716 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.200799 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:08.201124 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:38:08.201203 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:38:08.201259 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:08.201357 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 18:38:08.201408 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 18:38:08.202150 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:08.194211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40411 (local address 127.0.98.2:58887)
0619 18:38:08.194526 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:08.194542 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:08.194636 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:08.194952 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:08.194955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:08.194962 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 18:38:08.196570 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:08.196577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.197596 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.197600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:08.200276 (+ 2676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:08.200286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.200895 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.200898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:08.200960 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:08.201000 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 18:38:08.201281 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 18:38:08.201287 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:38:08.201341 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 18:38:08.201422 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":108,"threads_started":1}
I20260619 18:38:08.225221 7932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:38:08.225378 7932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:38:08.227324 7932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4: Sending DeleteTablet for 3 replicas of tablet 8792ceee5f454cb7a136dc5202ffb38e
I20260619 18:38:08.227923 8350 tablet_service.cc:1514] Processing DeleteTablet for tablet 8792ceee5f454cb7a136dc5202ffb38e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50074
I20260619 18:38:08.227953 8192 tablet_service.cc:1514] Processing DeleteTablet for tablet 8792ceee5f454cb7a136dc5202ffb38e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47772
I20260619 18:38:08.228060 8070 tablet_service.cc:1514] Processing DeleteTablet for tablet 8792ceee5f454cb7a136dc5202ffb38e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47842
I20260619 18:38:08.228116 8469 tablet_replica.cc:330] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96: stopping tablet replica
I20260619 18:38:08.228229 8469 raft_consensus.cc:2229] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:38:08.228348 8469 raft_consensus.cc:2258] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:08.228500 8470 tablet_replica.cc:330] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac: stopping tablet replica
I20260619 18:38:08.228605 8470 raft_consensus.cc:2229] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:38:08.228711 8470 raft_consensus.cc:2258] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:08.228852 8468 tablet_replica.cc:330] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8: stopping tablet replica
I20260619 18:38:08.228946 8468 raft_consensus.cc:2229] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:38:08.229238 8468 raft_consensus.cc:2258] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:08.230232 8469 ts_tablet_manager.cc:1905] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:08.233134 8469 ts_tablet_manager.cc:1918] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:38:08.233220 8469 log.cc:1196] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/8792ceee5f454cb7a136dc5202ffb38e
I20260619 18:38:08.233453 8469 ts_tablet_manager.cc:1939] T 8792ceee5f454cb7a136dc5202ffb38e P 7e2d75795ef64a8e80cef4598b29bf96: Deleting consensus metadata
I20260619 18:38:08.234143 7920 catalog_manager.cc:4841] TS 7e2d75795ef64a8e80cef4598b29bf96 (127.0.98.3:44115): tablet 8792ceee5f454cb7a136dc5202ffb38e (table test-table [id=98ab403e81094f999cbece1bf266ef09]) successfully deleted
I20260619 18:38:08.234935 8468 ts_tablet_manager.cc:1905] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:08.237653 8468 ts_tablet_manager.cc:1918] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:38:08.237721 8468 log.cc:1196] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/wal/wals/8792ceee5f454cb7a136dc5202ffb38e
I20260619 18:38:08.238615 8470 ts_tablet_manager.cc:1905] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:08.240541 8470 ts_tablet_manager.cc:1918] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:38:08.240607 8470 log.cc:1196] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/8792ceee5f454cb7a136dc5202ffb38e
I20260619 18:38:08.240792 8470 ts_tablet_manager.cc:1939] T 8792ceee5f454cb7a136dc5202ffb38e P 0686c947a5224e02b2eff73947972aac: Deleting consensus metadata
I20260619 18:38:08.241458 8468 ts_tablet_manager.cc:1939] T 8792ceee5f454cb7a136dc5202ffb38e P 0bb7854dc2714e08b1da22afdbae90c8: Deleting consensus metadata
I20260619 18:38:08.241458 7919 catalog_manager.cc:4841] TS 0686c947a5224e02b2eff73947972aac (127.0.98.2:42927): tablet 8792ceee5f454cb7a136dc5202ffb38e (table test-table [id=98ab403e81094f999cbece1bf266ef09]) successfully deleted
I20260619 18:38:08.242064 7917 catalog_manager.cc:4841] TS 0bb7854dc2714e08b1da22afdbae90c8 (127.0.98.1:40411): tablet 8792ceee5f454cb7a136dc5202ffb38e (table test-table [id=98ab403e81094f999cbece1bf266ef09]) successfully deleted
Jun 19 18:38:08 dist-test-slave-rxv6 krb5kdc[7888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 18:38:08 dist-test-slave-rxv6 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:08 dist-test-slave-rxv6 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:38:08.259439 7982 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:08.253052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58044 (local address 127.0.98.62:46527)
0619 18:38:08.253197 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:08.253202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:08.253428 (+ 226us) server_negotiation.cc:407] Connection header received
0619 18:38:08.253561 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:08.253565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:08.253619 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:08.253684 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:08.254468 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.255071 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.255953 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.256210 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.256301 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:08.259258 (+ 2957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58044: BlockingRecv error: recv got EOF from 127.0.0.1:58044 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 18:38:08 dist-test-slave-rxv6 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 18:38:08.268752 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:08.260767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58046 (local address 127.0.98.62:46527)
0619 18:38:08.261113 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:08.261116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:08.261498 (+ 382us) server_negotiation.cc:407] Connection header received
0619 18:38:08.261605 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:08.261608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:08.261650 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:08.261712 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:08.262353 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.262848 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.263605 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.263821 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.263902 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:08.265983 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:08.265987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:08.265990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:08.266012 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:08.267288 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:08.267655 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:08.267659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:08.267660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:08.267731 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:08.267961 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:08.267964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:08.267966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:08.268141 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:08.268254 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:08.268493 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 18:38:08.268597 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260619 18:38:08.271400 7932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:38:08.271829 7932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:38:08.276535 8070 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7b7c1936cc4b888727402667a1d9ef (DEFAULT_TABLE table=test-table [id=a595e01a6fa44bbcabaf269f6715179f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:08.276845 8070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7b7c1936cc4b888727402667a1d9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:08.276820 8192 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7b7c1936cc4b888727402667a1d9ef (DEFAULT_TABLE table=test-table [id=a595e01a6fa44bbcabaf269f6715179f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:08.277096 8192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7b7c1936cc4b888727402667a1d9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:08.278972 8446 tablet_bootstrap.cc:492] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8: Bootstrap starting.
I20260619 18:38:08.279209 8350 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7b7c1936cc4b888727402667a1d9ef (DEFAULT_TABLE table=test-table [id=a595e01a6fa44bbcabaf269f6715179f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:08.279435 8444 tablet_bootstrap.cc:492] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac: Bootstrap starting.
I20260619 18:38:08.279460 8350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7b7c1936cc4b888727402667a1d9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:08.279930 8446 tablet_bootstrap.cc:654] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:08.280180 8444 tablet_bootstrap.cc:654] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:08.281575 8446 tablet_bootstrap.cc:492] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8: No bootstrap required, opened a new log
I20260619 18:38:08.281598 8445 tablet_bootstrap.cc:492] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96: Bootstrap starting.
I20260619 18:38:08.281659 8446 ts_tablet_manager.cc:1397] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 18:38:08.282114 8446 raft_consensus.cc:348] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.282235 8446 raft_consensus.cc:374] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:08.282267 8446 raft_consensus.cc:729] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb7854dc2714e08b1da22afdbae90c8, State: Initialized, Role: FOLLOWER
I20260619 18:38:08.282378 8444 tablet_bootstrap.cc:492] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac: No bootstrap required, opened a new log
I20260619 18:38:08.282387 8446 consensus_queue.cc:260] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.282451 8444 ts_tablet_manager.cc:1397] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 18:38:08.282578 8445 tablet_bootstrap.cc:654] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:08.282631 8446 ts_tablet_manager.cc:1428] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 18:38:08.282824 8444 raft_consensus.cc:348] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.282922 8444 raft_consensus.cc:374] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:08.282951 8444 raft_consensus.cc:729] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0686c947a5224e02b2eff73947972aac, State: Initialized, Role: FOLLOWER
I20260619 18:38:08.283067 8444 consensus_queue.cc:260] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.283301 8444 ts_tablet_manager.cc:1428] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 18:38:08.284137 8445 tablet_bootstrap.cc:492] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96: No bootstrap required, opened a new log
I20260619 18:38:08.284219 8445 ts_tablet_manager.cc:1397] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 18:38:08.284641 8445 raft_consensus.cc:348] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.284737 8445 raft_consensus.cc:374] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:08.284766 8445 raft_consensus.cc:729] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e2d75795ef64a8e80cef4598b29bf96, State: Initialized, Role: FOLLOWER
I20260619 18:38:08.284879 8445 consensus_queue.cc:260] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.285159 8445 ts_tablet_manager.cc:1428] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 18:38:08.389254 8451 raft_consensus.cc:482] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:38:08.389374 8451 raft_consensus.cc:504] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.390453 8451 leader_election.cc:288] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bb7854dc2714e08b1da22afdbae90c8 (127.0.98.1:40411), 0686c947a5224e02b2eff73947972aac (127.0.98.2:42927)
I20260619 18:38:08.390928 8498 raft_consensus.cc:482] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:38:08.391026 8498 raft_consensus.cc:504] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.391346 8498 leader_election.cc:288] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0686c947a5224e02b2eff73947972aac (127.0.98.2:42927), 7e2d75795ef64a8e80cef4598b29bf96 (127.0.98.3:44115)
I20260619 18:38:08.391902 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7b7c1936cc4b888727402667a1d9ef" candidate_uuid: "0bb7854dc2714e08b1da22afdbae90c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0686c947a5224e02b2eff73947972aac" is_pre_election: true
I20260619 18:38:08.392055 8230 raft_consensus.cc:2454] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bb7854dc2714e08b1da22afdbae90c8 in term 0.
I20260619 18:38:08.392122 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7b7c1936cc4b888727402667a1d9ef" candidate_uuid: "0bb7854dc2714e08b1da22afdbae90c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e2d75795ef64a8e80cef4598b29bf96" is_pre_election: true
I20260619 18:38:08.392241 8370 raft_consensus.cc:2454] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bb7854dc2714e08b1da22afdbae90c8 in term 0.
I20260619 18:38:08.392369 8023 leader_election.cc:302] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0686c947a5224e02b2eff73947972aac, 0bb7854dc2714e08b1da22afdbae90c8; no voters:
I20260619 18:38:08.392606 8498 raft_consensus.cc:2781] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:38:08.392656 8498 raft_consensus.cc:482] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:38:08.392690 8498 raft_consensus.cc:3037] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:08.393805 8498 raft_consensus.cc:504] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.394129 8498 leader_election.cc:288] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [CANDIDATE]: Term 1 election: Requested vote from peers 0686c947a5224e02b2eff73947972aac (127.0.98.2:42927), 7e2d75795ef64a8e80cef4598b29bf96 (127.0.98.3:44115)
I20260619 18:38:08.394495 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7b7c1936cc4b888727402667a1d9ef" candidate_uuid: "0bb7854dc2714e08b1da22afdbae90c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0686c947a5224e02b2eff73947972aac"
I20260619 18:38:08.394622 8230 raft_consensus.cc:3037] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:08.395658 8230 raft_consensus.cc:2454] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bb7854dc2714e08b1da22afdbae90c8 in term 1.
I20260619 18:38:08.395941 8023 leader_election.cc:302] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0686c947a5224e02b2eff73947972aac, 0bb7854dc2714e08b1da22afdbae90c8; no voters:
I20260619 18:38:08.396230 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7b7c1936cc4b888727402667a1d9ef" candidate_uuid: "0bb7854dc2714e08b1da22afdbae90c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e2d75795ef64a8e80cef4598b29bf96"
I20260619 18:38:08.396323 8370 raft_consensus.cc:3037] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:08.396417 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:08.390831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60905 (local address 127.0.98.1:40411)
0619 18:38:08.391131 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:08.391135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:08.391337 (+ 202us) server_negotiation.cc:407] Connection header received
0619 18:38:08.391485 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:08.391489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:08.391534 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:08.391579 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:08.392849 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.393532 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.395048 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.395638 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.395721 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:08.395773 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:08.396232 (+ 459us) server_negotiation.cc:299] Negotiation successful
0619 18:38:08.396304 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222}
I20260619 18:38:08.396417 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:08.390781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48935 (local address 127.0.98.2:42927)
0619 18:38:08.390901 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:08.390905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:08.391293 (+ 388us) server_negotiation.cc:407] Connection header received
0619 18:38:08.391538 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:08.391541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:08.391606 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:08.391648 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:08.392385 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.393096 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.395048 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.395638 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.395710 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:08.395770 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:08.396234 (+ 464us) server_negotiation.cc:299] Negotiation successful
0619 18:38:08.396304 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:38:08.396817 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:08.390687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40411 (local address 127.0.98.3:60905)
0619 18:38:08.391276 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:08.391287 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:08.391380 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:08.392574 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:08.392577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:08.392595 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:08.392738 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:08.392745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.393687 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.393690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:08.394912 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:08.394922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.395882 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.395886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:08.396111 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:08.396129 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 18:38:08.396575 (+ 446us) client_negotiation.cc:240] Negotiation successful
0619 18:38:08.396653 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":319,"thread_start_us":60,"threads_started":1}
I20260619 18:38:08.396885 8498 raft_consensus.cc:2781] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:38:08.396963 8498 raft_consensus.cc:686] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0bb7854dc2714e08b1da22afdbae90c8, State: Running, Role: LEADER
I20260619 18:38:08.397094 8498 consensus_queue.cc:237] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.397209 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7b7c1936cc4b888727402667a1d9ef" candidate_uuid: "7e2d75795ef64a8e80cef4598b29bf96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bb7854dc2714e08b1da22afdbae90c8" is_pre_election: true
I20260619 18:38:08.397257 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:08.390687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42927 (local address 127.0.98.3:48935)
0619 18:38:08.391192 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:08.391218 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:08.391374 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:08.391875 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:08.391877 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:08.391895 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:08.392094 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:08.392103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.393321 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.393324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:08.394919 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:08.394925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.395892 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.395895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:08.396066 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:08.396093 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 18:38:08.396752 (+ 659us) client_negotiation.cc:240] Negotiation successful
0619 18:38:08.396813 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":6,"thread_start_us":67,"threads_started":1}
I20260619 18:38:08.397289 8370 raft_consensus.cc:2454] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bb7854dc2714e08b1da22afdbae90c8 in term 1.
I20260619 18:38:08.397792 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7b7c1936cc4b888727402667a1d9ef" candidate_uuid: "7e2d75795ef64a8e80cef4598b29bf96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0686c947a5224e02b2eff73947972aac" is_pre_election: true
I20260619 18:38:08.397991 8230 raft_consensus.cc:2379] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7e2d75795ef64a8e80cef4598b29bf96 in current term 1: Already voted for candidate 0bb7854dc2714e08b1da22afdbae90c8 in this term.
I20260619 18:38:08.398294 8303 leader_election.cc:302] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7e2d75795ef64a8e80cef4598b29bf96; no voters: 0686c947a5224e02b2eff73947972aac, 0bb7854dc2714e08b1da22afdbae90c8
I20260619 18:38:08.398695 8451 raft_consensus.cc:2726] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 18:38:08.399943 7932 catalog_manager.cc:5495] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bb7854dc2714e08b1da22afdbae90c8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0bb7854dc2714e08b1da22afdbae90c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } health_report { overall_health: UNKNOWN } } }
I20260619 18:38:08.410815 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:08.407600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47862 (local address 127.0.98.1:40411)
0619 18:38:08.407714 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:08.407717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:08.407731 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:38:08.407846 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:08.407849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:08.407890 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:08.407952 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:38:08.408162 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.408665 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.409841 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:08.410116 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:08.410200 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:08.410447 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:38:08.410523 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:38:08.410575 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:08.410664 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 18:38:08.410708 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:38:08.412837 8230 raft_consensus.cc:1261] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Refusing update from remote peer 0bb7854dc2714e08b1da22afdbae90c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:38:08.412904 8370 raft_consensus.cc:1261] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Refusing update from remote peer 0bb7854dc2714e08b1da22afdbae90c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:38:08.413326 8498 consensus_queue.cc:1035] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:38:08.413482 8452 consensus_queue.cc:1035] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:38:08.420753 7933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:38:08.420910 7933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:38:08.422771 7933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4: Sending DeleteTablet for 3 replicas of tablet 3e7b7c1936cc4b888727402667a1d9ef
I20260619 18:38:08.423306 8070 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e7b7c1936cc4b888727402667a1d9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47842
I20260619 18:38:08.423336 8350 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e7b7c1936cc4b888727402667a1d9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50074
I20260619 18:38:08.423506 8468 tablet_replica.cc:330] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8: stopping tablet replica
I20260619 18:38:08.423516 8192 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e7b7c1936cc4b888727402667a1d9ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47772
I20260619 18:38:08.423614 8468 raft_consensus.cc:2229] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:38:08.423666 8470 tablet_replica.cc:330] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac: stopping tablet replica
I20260619 18:38:08.423759 8470 raft_consensus.cc:2229] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:38:08.423802 8468 raft_consensus.cc:2258] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:08.423879 8470 raft_consensus.cc:2258] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:08.423967 8469 tablet_replica.cc:330] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96: stopping tablet replica
I20260619 18:38:08.424082 8469 raft_consensus.cc:2229] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:38:08.424212 8469 raft_consensus.cc:2258] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:08.424512 8470 ts_tablet_manager.cc:1905] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:08.424950 8469 ts_tablet_manager.cc:1905] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:08.424979 8468 ts_tablet_manager.cc:1905] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:08.426829 8468 ts_tablet_manager.cc:1918] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:38:08.426882 8468 log.cc:1196] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-0/wal/wals/3e7b7c1936cc4b888727402667a1d9ef
I20260619 18:38:08.426990 8470 ts_tablet_manager.cc:1918] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:38:08.427042 8468 ts_tablet_manager.cc:1939] T 3e7b7c1936cc4b888727402667a1d9ef P 0bb7854dc2714e08b1da22afdbae90c8: Deleting consensus metadata
I20260619 18:38:08.427060 8470 log.cc:1196] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/3e7b7c1936cc4b888727402667a1d9ef
I20260619 18:38:08.427219 8469 ts_tablet_manager.cc:1918] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:38:08.427279 8469 log.cc:1196] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/3e7b7c1936cc4b888727402667a1d9ef
I20260619 18:38:08.427277 8470 ts_tablet_manager.cc:1939] T 3e7b7c1936cc4b888727402667a1d9ef P 0686c947a5224e02b2eff73947972aac: Deleting consensus metadata
I20260619 18:38:08.427485 8469 ts_tablet_manager.cc:1939] T 3e7b7c1936cc4b888727402667a1d9ef P 7e2d75795ef64a8e80cef4598b29bf96: Deleting consensus metadata
I20260619 18:38:08.427554 7933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:38:08.427907 7933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:38:08.428158 7919 catalog_manager.cc:4841] TS 0686c947a5224e02b2eff73947972aac (127.0.98.2:42927): tablet 3e7b7c1936cc4b888727402667a1d9ef (table test-table [id=a595e01a6fa44bbcabaf269f6715179f]) successfully deleted
I20260619 18:38:08.428483 7917 catalog_manager.cc:4841] TS 0bb7854dc2714e08b1da22afdbae90c8 (127.0.98.1:40411): tablet 3e7b7c1936cc4b888727402667a1d9ef (table test-table [id=a595e01a6fa44bbcabaf269f6715179f]) successfully deleted
I20260619 18:38:08.429088 7920 catalog_manager.cc:4841] TS 7e2d75795ef64a8e80cef4598b29bf96 (127.0.98.3:44115): tablet 3e7b7c1936cc4b888727402667a1d9ef (table test-table [id=a595e01a6fa44bbcabaf269f6715179f]) successfully deleted
I20260619 18:38:08.432098 8070 tablet_service.cc:1467] Processing CreateTablet for tablet cfeb441d4d12435dba2443583ef64ac2 (DEFAULT_TABLE table=test-table [id=b03c9f4bf0f34bcba8977f40ee79b854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:08.432130 8192 tablet_service.cc:1467] Processing CreateTablet for tablet cfeb441d4d12435dba2443583ef64ac2 (DEFAULT_TABLE table=test-table [id=b03c9f4bf0f34bcba8977f40ee79b854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:08.432350 8070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfeb441d4d12435dba2443583ef64ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:08.432368 8192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfeb441d4d12435dba2443583ef64ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:08.434141 8446 tablet_bootstrap.cc:492] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8: Bootstrap starting.
I20260619 18:38:08.434238 8444 tablet_bootstrap.cc:492] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac: Bootstrap starting.
I20260619 18:38:08.434304 8350 tablet_service.cc:1467] Processing CreateTablet for tablet cfeb441d4d12435dba2443583ef64ac2 (DEFAULT_TABLE table=test-table [id=b03c9f4bf0f34bcba8977f40ee79b854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:08.434547 8350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfeb441d4d12435dba2443583ef64ac2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:08.434969 8444 tablet_bootstrap.cc:654] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:08.434969 8446 tablet_bootstrap.cc:654] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:08.436219 8446 tablet_bootstrap.cc:492] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8: No bootstrap required, opened a new log
I20260619 18:38:08.436244 8444 tablet_bootstrap.cc:492] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac: No bootstrap required, opened a new log
I20260619 18:38:08.436305 8446 ts_tablet_manager.cc:1397] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:38:08.436305 8444 ts_tablet_manager.cc:1397] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:38:08.436669 8446 raft_consensus.cc:348] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.436669 8444 raft_consensus.cc:348] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.436784 8446 raft_consensus.cc:374] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:08.436784 8444 raft_consensus.cc:374] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:08.436825 8444 raft_consensus.cc:729] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0686c947a5224e02b2eff73947972aac, State: Initialized, Role: FOLLOWER
I20260619 18:38:08.436825 8446 raft_consensus.cc:729] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb7854dc2714e08b1da22afdbae90c8, State: Initialized, Role: FOLLOWER
I20260619 18:38:08.436942 8446 consensus_queue.cc:260] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.436942 8444 consensus_queue.cc:260] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.437188 8444 ts_tablet_manager.cc:1428] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:38:08.437201 8446 ts_tablet_manager.cc:1428] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:38:08.437390 8445 tablet_bootstrap.cc:492] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96: Bootstrap starting.
I20260619 18:38:08.438128 8445 tablet_bootstrap.cc:654] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:08.439273 8445 tablet_bootstrap.cc:492] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96: No bootstrap required, opened a new log
I20260619 18:38:08.439353 8445 ts_tablet_manager.cc:1397] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 18:38:08.439770 8445 raft_consensus.cc:348] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.439857 8445 raft_consensus.cc:374] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:08.439875 8445 raft_consensus.cc:729] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e2d75795ef64a8e80cef4598b29bf96, State: Initialized, Role: FOLLOWER
I20260619 18:38:08.439970 8445 consensus_queue.cc:260] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.440183 8445 ts_tablet_manager.cc:1428] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 18:38:08.539546 8498 raft_consensus.cc:482] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:38:08.539681 8498 raft_consensus.cc:504] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.540124 8498 leader_election.cc:288] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0686c947a5224e02b2eff73947972aac (127.0.98.2:42927), 7e2d75795ef64a8e80cef4598b29bf96 (127.0.98.3:44115)
I20260619 18:38:08.540550 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeb441d4d12435dba2443583ef64ac2" candidate_uuid: "0bb7854dc2714e08b1da22afdbae90c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0686c947a5224e02b2eff73947972aac" is_pre_election: true
I20260619 18:38:08.540575 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeb441d4d12435dba2443583ef64ac2" candidate_uuid: "0bb7854dc2714e08b1da22afdbae90c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e2d75795ef64a8e80cef4598b29bf96" is_pre_election: true
I20260619 18:38:08.540722 8370 raft_consensus.cc:2454] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bb7854dc2714e08b1da22afdbae90c8 in term 0.
I20260619 18:38:08.540719 8230 raft_consensus.cc:2454] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0bb7854dc2714e08b1da22afdbae90c8 in term 0.
I20260619 18:38:08.540998 8024 leader_election.cc:302] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb7854dc2714e08b1da22afdbae90c8, 7e2d75795ef64a8e80cef4598b29bf96; no voters:
I20260619 18:38:08.541175 8498 raft_consensus.cc:2781] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:38:08.541231 8498 raft_consensus.cc:482] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:38:08.541257 8498 raft_consensus.cc:3037] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:08.542273 8498 raft_consensus.cc:504] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.542582 8498 leader_election.cc:288] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [CANDIDATE]: Term 1 election: Requested vote from peers 0686c947a5224e02b2eff73947972aac (127.0.98.2:42927), 7e2d75795ef64a8e80cef4598b29bf96 (127.0.98.3:44115)
I20260619 18:38:08.542900 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeb441d4d12435dba2443583ef64ac2" candidate_uuid: "0bb7854dc2714e08b1da22afdbae90c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0686c947a5224e02b2eff73947972aac"
I20260619 18:38:08.542919 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfeb441d4d12435dba2443583ef64ac2" candidate_uuid: "0bb7854dc2714e08b1da22afdbae90c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e2d75795ef64a8e80cef4598b29bf96"
I20260619 18:38:08.543006 8230 raft_consensus.cc:3037] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:08.543031 8370 raft_consensus.cc:3037] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:08.543869 8370 raft_consensus.cc:2454] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bb7854dc2714e08b1da22afdbae90c8 in term 1.
I20260619 18:38:08.543937 8230 raft_consensus.cc:2454] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bb7854dc2714e08b1da22afdbae90c8 in term 1.
I20260619 18:38:08.544138 8024 leader_election.cc:302] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bb7854dc2714e08b1da22afdbae90c8, 7e2d75795ef64a8e80cef4598b29bf96; no voters:
I20260619 18:38:08.544303 8498 raft_consensus.cc:2781] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:38:08.544421 8498 raft_consensus.cc:686] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 LEADER]: Becoming Leader. State: Replica: 0bb7854dc2714e08b1da22afdbae90c8, State: Running, Role: LEADER
I20260619 18:38:08.544545 8498 consensus_queue.cc:237] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } }
I20260619 18:38:08.546149 7930 catalog_manager.cc:5495] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0bb7854dc2714e08b1da22afdbae90c8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "0bb7854dc2714e08b1da22afdbae90c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bb7854dc2714e08b1da22afdbae90c8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 } health_report { overall_health: UNKNOWN } } }
I20260619 18:38:08.567847 8370 raft_consensus.cc:1261] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Refusing update from remote peer 0bb7854dc2714e08b1da22afdbae90c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:38:08.567947 8230 raft_consensus.cc:1261] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Refusing update from remote peer 0bb7854dc2714e08b1da22afdbae90c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:38:08.568207 8476 consensus_queue.cc:1035] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e2d75795ef64a8e80cef4598b29bf96" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:38:08.568353 8498 consensus_queue.cc:1035] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0686c947a5224e02b2eff73947972aac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:38:08.596485 7930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:38:08.596638 7930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:38:08.598568 7930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f989fe2fe0e844c1b6f75886d9b3e1c4: Sending DeleteTablet for 3 replicas of tablet cfeb441d4d12435dba2443583ef64ac2
I20260619 18:38:08.599203 8350 tablet_service.cc:1514] Processing DeleteTablet for tablet cfeb441d4d12435dba2443583ef64ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50074
I20260619 18:38:08.599264 8070 tablet_service.cc:1514] Processing DeleteTablet for tablet cfeb441d4d12435dba2443583ef64ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47842
I20260619 18:38:08.599292 8192 tablet_service.cc:1514] Processing DeleteTablet for tablet cfeb441d4d12435dba2443583ef64ac2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:08 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47772
I20260619 18:38:08.599393 8469 tablet_replica.cc:330] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96: stopping tablet replica
I20260619 18:38:08.599403 8468 tablet_replica.cc:330] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8: stopping tablet replica
I20260619 18:38:08.599501 8468 raft_consensus.cc:2229] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:38:08.599498 8469 raft_consensus.cc:2229] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:38:08.599617 8469 raft_consensus.cc:2258] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:08.599658 8470 tablet_replica.cc:330] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac: stopping tablet replica
I20260619 18:38:08.599747 8468 raft_consensus.cc:2258] T cfeb441d4d12435dba2443583ef64ac2 P 0bb7854dc2714e08b1da22afdbae90c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:08.599776 8470 raft_consensus.cc:2229] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:38:08.599884 8470 raft_consensus.cc:2258] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:08.599903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 8005
I20260619 18:38:08.600777 8470 ts_tablet_manager.cc:1905] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:08.602044 8469 ts_tablet_manager.cc:1905] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:08.603304 8470 ts_tablet_manager.cc:1918] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:38:08.603390 8470 log.cc:1196] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/cfeb441d4d12435dba2443583ef64ac2
I20260619 18:38:08.603631 8470 ts_tablet_manager.cc:1939] T cfeb441d4d12435dba2443583ef64ac2 P 0686c947a5224e02b2eff73947972aac: Deleting consensus metadata
I20260619 18:38:08.604362 8469 ts_tablet_manager.cc:1918] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 18:38:08.604439 8469 log.cc:1196] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/cfeb441d4d12435dba2443583ef64ac2
I20260619 18:38:08.604425 7919 catalog_manager.cc:4841] TS 0686c947a5224e02b2eff73947972aac (127.0.98.2:42927): tablet cfeb441d4d12435dba2443583ef64ac2 (table test-table [id=b03c9f4bf0f34bcba8977f40ee79b854]) successfully deleted
I20260619 18:38:08.604663 8469 ts_tablet_manager.cc:1939] T cfeb441d4d12435dba2443583ef64ac2 P 7e2d75795ef64a8e80cef4598b29bf96: Deleting consensus metadata
I20260619 18:38:08.605286 7920 catalog_manager.cc:4841] TS 7e2d75795ef64a8e80cef4598b29bf96 (127.0.98.3:44115): tablet cfeb441d4d12435dba2443583ef64ac2 (table test-table [id=b03c9f4bf0f34bcba8977f40ee79b854]) successfully deleted
I20260619 18:38:08.608310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 8145
W20260619 18:38:08.608275 7917 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40411 (error 108)
I20260619 18:38:08.610261 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:08.609728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40411 (local address 127.0.0.1:47872)
0619 18:38:08.609995 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:08.610100 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":62,"threads_started":1}
W20260619 18:38:08.610414 7917 catalog_manager.cc:4584] TS 0bb7854dc2714e08b1da22afdbae90c8 (127.0.98.1:40411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cfeb441d4d12435dba2443583ef64ac2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40411: connect: Connection refused (error 111)
I20260619 18:38:08.614782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 8285
I20260619 18:38:08.619756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 7901
2026-06-19T18:38:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3285 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:38:08 dist-test-slave-rxv6 krb5kdc[8536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:38:08 dist-test-slave-rxv6 krb5kdc[8536](info): set up 2 sockets
Jun 19 18:38:08 dist-test-slave-rxv6 krb5kdc[8536](info): commencing operation
krb5kdc: starting...
W20260619 18:38:09.582481 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:38:09 dist-test-slave-rxv6 krb5kdc[8536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:38:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:38:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:38:09.703116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42167
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:09.814028 8549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:09.814307 8549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:09.814347 8549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:09.818362 8549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:38:09.818424 8549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:09.818442 8549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:09.818457 8549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:38:09.818470 8549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:38:09.823809 8549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:09.824808 8549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:09.825793 8549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:09.834087 8554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:09.834117 8555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:09.834079 8557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:09.834313 8549 server_base.cc:1034] running on GCE node
I20260619 18:38:09.835045 8549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:09.836153 8549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:09.837332 8549 hybrid_clock.cc:648] HybridClock initialized: now 1781894289837318 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T18:38:09Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1221 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:38:09 dist-test-slave-rxv6 krb5kdc[8562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:38:09 dist-test-slave-rxv6 krb5kdc[8562](info): set up 2 sockets
Jun 19 18:38:09 dist-test-slave-rxv6 krb5kdc[8562](info): commencing operation
krb5kdc: starting...
W20260619 18:38:10.802440 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:38:10 dist-test-slave-rxv6 krb5kdc[8562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:38:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:38:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:38:10.920459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35899
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:11.026782 8575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:11.027086 8575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:11.027144 8575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:11.031318 8575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:38:11.031391 8575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:11.031418 8575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:11.031441 8575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:38:11.031467 8575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:38:11.036893 8575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:11.037998 8575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:11.038981 8575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:11.047032 8581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:11.047101 8580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:11.047092 8575 server_base.cc:1034] running on GCE node
W20260619 18:38:11.047040 8583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:11.047803 8575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:11.048868 8575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:11.049989 8575 hybrid_clock.cc:648] HybridClock initialized: now 1781894291049963 us; error 40 us; skew 500 ppm
Jun 19 18:38:11 dist-test-slave-rxv6 krb5kdc[8562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:11.052551 8575 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:38:11.053397 8575 webserver.cc:460] Webserver started at http://127.0.98.62:41181/ using document root <none> and password file <none>
I20260619 18:38:11.053938 8575 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:11.054009 8575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:11.054198 8575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:11.056178 8575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "1850f66ec25a4d64b56de89dadb15358"
format_stamp: "Formatted at 2026-06-19 18:38:11 on dist-test-slave-rxv6"
server_key: "317c00aa5703abed2f8a4c2b6d01908a"
server_key_iv: "a2dd3cf805ab0c32be84a3ac6e554491"
server_key_version: "encryptionkey@0"
I20260619 18:38:11.056682 8575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "1850f66ec25a4d64b56de89dadb15358"
format_stamp: "Formatted at 2026-06-19 18:38:11 on dist-test-slave-rxv6"
server_key: "317c00aa5703abed2f8a4c2b6d01908a"
server_key_iv: "a2dd3cf805ab0c32be84a3ac6e554491"
server_key_version: "encryptionkey@0"
I20260619 18:38:11.060108 8575 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:38:11.062289 8590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:11.063422 8575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:38:11.063526 8575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "1850f66ec25a4d64b56de89dadb15358"
format_stamp: "Formatted at 2026-06-19 18:38:11 on dist-test-slave-rxv6"
server_key: "317c00aa5703abed2f8a4c2b6d01908a"
server_key_iv: "a2dd3cf805ab0c32be84a3ac6e554491"
server_key_version: "encryptionkey@0"
I20260619 18:38:11.063635 8575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:11.079051 8575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:11.082219 8575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:11.082396 8575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:11.088968 8575 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35899
I20260619 18:38:11.089013 8642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35899 every 8 connection(s)
I20260619 18:38:11.089933 8575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:38:11.092656 8643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:11.096236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 8575
I20260619 18:38:11.096336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:38:11.096588 392 external_mini_cluster.cc:1428] Setting key 1b562a807d2981c705a06601472bbaa0
I20260619 18:38:11.098361 8643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358: Bootstrap starting.
I20260619 18:38:11.101033 8643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:11.101964 8643 log.cc:824] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:11.104094 8643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358: No bootstrap required, opened a new log
Jun 19 18:38:11 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894290, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:38:11.108314 8643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1850f66ec25a4d64b56de89dadb15358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35899 } }
I20260619 18:38:11.108601 8643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:11.108662 8643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1850f66ec25a4d64b56de89dadb15358, State: Initialized, Role: FOLLOWER
I20260619 18:38:11.109177 8643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1850f66ec25a4d64b56de89dadb15358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35899 } }
I20260619 18:38:11.109352 8643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:38:11.109422 8643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:38:11.109524 8643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:11.110692 8643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1850f66ec25a4d64b56de89dadb15358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35899 } }
I20260619 18:38:11.111055 8643 leader_election.cc:302] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1850f66ec25a4d64b56de89dadb15358; no voters:
I20260619 18:38:11.111477 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:11.097928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43122 (local address 127.0.98.62:35899)
0619 18:38:11.098345 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:11.098365 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:11.098408 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:38:11.099208 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:11.099231 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:11.099662 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:11.100045 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:11.100672 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:11.102197 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:11.103188 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:11.103836 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:11.103991 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:11.106335 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:11.106342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:11.106360 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:11.106394 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:11.108316 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:11.108747 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:11.108753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:11.108759 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:11.108898 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:11.109164 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:11.109167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:11.109168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:11.109732 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:11.109962 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:11.110168 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 18:38:11.110469 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":115,"threads_started":1}
I20260619 18:38:11.111420 8643 leader_election.cc:288] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:38:11.111588 8648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:38:11.111804 8648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [term 1 LEADER]: Becoming Leader. State: Replica: 1850f66ec25a4d64b56de89dadb15358, State: Running, Role: LEADER
I20260619 18:38:11.112176 8648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1850f66ec25a4d64b56de89dadb15358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35899 } }
I20260619 18:38:11.112794 8643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:38:11.113763 8649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1850f66ec25a4d64b56de89dadb15358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1850f66ec25a4d64b56de89dadb15358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35899 } } }
I20260619 18:38:11.114200 8649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:11.114094 8650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1850f66ec25a4d64b56de89dadb15358. Latest consensus state: current_term: 1 leader_uuid: "1850f66ec25a4d64b56de89dadb15358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1850f66ec25a4d64b56de89dadb15358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35899 } } }
I20260619 18:38:11.114542 8650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:11.114607 8660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:38:11.117409 8660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:38:11.122604 8660 catalog_manager.cc:1347] Generated new cluster ID: 8f74a76e6f2a4d26a4f148c18636658a
I20260619 18:38:11.122699 8660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:38:11.154752 8660 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:38:11.156050 8660 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:38:11.161001 8660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358: Generated new TSK 0
I20260619 18:38:11.161685 8660 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:38:11.197144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35899
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:11.318133 8669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:11.318411 8669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:11.318454 8669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:11.323030 8669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:11.323102 8669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:11.323163 8669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:38:11.328894 8669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:11.330067 8669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:11.331418 8669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:11.340929 8677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:11.341049 8669 server_base.cc:1034] running on GCE node
W20260619 18:38:11.340938 8674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:11.340950 8675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:11.341677 8669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:11.342355 8669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:11.343536 8669 hybrid_clock.cc:648] HybridClock initialized: now 1781894291343522 us; error 26 us; skew 500 ppm
Jun 19 18:38:11 dist-test-slave-rxv6 krb5kdc[8562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:11.346154 8669 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:38:11.347013 8669 webserver.cc:460] Webserver started at http://127.0.98.1:45077/ using document root <none> and password file <none>
I20260619 18:38:11.347532 8669 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:11.347587 8669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:11.347754 8669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:11.349855 8669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "076732515a434d03be9469d49448f9c7"
format_stamp: "Formatted at 2026-06-19 18:38:11 on dist-test-slave-rxv6"
server_key: "62918d9d2cd85643902d318f14b150b7"
server_key_iv: "41abf153cc2fb2ff54e4ac2d3fb22f00"
server_key_version: "encryptionkey@0"
I20260619 18:38:11.350414 8669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "076732515a434d03be9469d49448f9c7"
format_stamp: "Formatted at 2026-06-19 18:38:11 on dist-test-slave-rxv6"
server_key: "62918d9d2cd85643902d318f14b150b7"
server_key_iv: "41abf153cc2fb2ff54e4ac2d3fb22f00"
server_key_version: "encryptionkey@0"
I20260619 18:38:11.354064 8669 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:38:11.356346 8684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:11.357712 8669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:38:11.357810 8669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "076732515a434d03be9469d49448f9c7"
format_stamp: "Formatted at 2026-06-19 18:38:11 on dist-test-slave-rxv6"
server_key: "62918d9d2cd85643902d318f14b150b7"
server_key_iv: "41abf153cc2fb2ff54e4ac2d3fb22f00"
server_key_version: "encryptionkey@0"
I20260619 18:38:11.357941 8669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:11.404086 8669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:11.407467 8669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:11.407661 8669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:11.408280 8669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:11.409135 8669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:11.409183 8669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:11.409224 8669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:11.409250 8669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:11.418193 8669 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40265
I20260619 18:38:11.418233 8797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40265 every 8 connection(s)
I20260619 18:38:11.419112 8669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:38:11.424758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 8669
I20260619 18:38:11.424885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:38:11.425156 392 external_mini_cluster.cc:1428] Setting key 48bba7b706f27c69ba071ba53e9b7a9d
Jun 19 18:38:11 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:38:11.433444 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:11.421103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55553 (local address 127.0.98.62:35899)
0619 18:38:11.421245 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:11.421250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:11.422175 (+ 925us) server_negotiation.cc:407] Connection header received
0619 18:38:11.423493 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:11.423496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:11.423543 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:11.423620 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:11.424997 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:11.425607 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:11.427050 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:11.427279 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:11.427365 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:11.430128 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:11.430131 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:11.430133 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:11.430158 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:11.431490 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:11.431985 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:11.431988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:11.431989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:11.432069 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:11.432379 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:11.432383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:11.432385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:11.432560 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:11.432641 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:11.433144 (+ 503us) server_negotiation.cc:299] Negotiation successful
0619 18:38:11.433268 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 18:38:11.434322 8800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:11.421524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:55553)
0619 18:38:11.422013 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:11.422074 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:11.423296 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:11.423791 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:11.423798 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:11.424180 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:11.424816 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:11.424839 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:11.425737 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:11.425741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:11.426929 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:11.426940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:11.427424 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:11.427428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:11.427634 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:11.428294 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:11.428312 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:11.429962 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:11.431604 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:11.431611 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:11.431628 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:11.431883 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:11.432146 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:11.432149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:11.432151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:11.432288 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:11.432651 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:11.432658 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:11.432906 (+ 248us) client_negotiation.cc:769] Sending connection context
0619 18:38:11.433066 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 18:38:11.433400 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":110,"threads_started":1}
I20260619 18:38:11.435516 8798 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35899
I20260619 18:38:11.435767 8798 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 18:38:11.436297 8798 heartbeater.cc:511] Master 127.0.98.62:35899 requested a full tablet report, sending...
I20260619 18:38:11.437790 8600 ts_manager.cc:194] Registered new tserver with Master: 076732515a434d03be9469d49448f9c7 (127.0.98.1:40265)
I20260619 18:38:11.438900 8600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:38:11.451268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35899
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:11.561877 8803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:11.562216 8803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:11.562273 8803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:11.566524 8803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:11.566596 8803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:11.566656 8803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:38:11.572069 8803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:11.573148 8803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:11.574142 8803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:11.583945 8808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:11.583959 8809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:11.584148 8803 server_base.cc:1034] running on GCE node
W20260619 18:38:11.583977 8811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:11.584679 8803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:11.585506 8803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:11.586689 8803 hybrid_clock.cc:648] HybridClock initialized: now 1781894291586661 us; error 38 us; skew 500 ppm
Jun 19 18:38:11 dist-test-slave-rxv6 krb5kdc[8562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:11.592778 8803 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:38:11.593705 8803 webserver.cc:460] Webserver started at http://127.0.98.2:38013/ using document root <none> and password file <none>
I20260619 18:38:11.594307 8803 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:11.594370 8803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:11.594555 8803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:11.596591 8803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "31a3b574fed342bf88ac3c532d231fb6"
format_stamp: "Formatted at 2026-06-19 18:38:11 on dist-test-slave-rxv6"
server_key: "de5f0fb63e02f62e1d0a5324bee7794d"
server_key_iv: "7b7b31eb8efcca8b735f8f897693b42b"
server_key_version: "encryptionkey@0"
I20260619 18:38:11.597106 8803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "31a3b574fed342bf88ac3c532d231fb6"
format_stamp: "Formatted at 2026-06-19 18:38:11 on dist-test-slave-rxv6"
server_key: "de5f0fb63e02f62e1d0a5324bee7794d"
server_key_iv: "7b7b31eb8efcca8b735f8f897693b42b"
server_key_version: "encryptionkey@0"
I20260619 18:38:11.600725 8803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 18:38:11.603122 8818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:11.604328 8803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:38:11.604423 8803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "31a3b574fed342bf88ac3c532d231fb6"
format_stamp: "Formatted at 2026-06-19 18:38:11 on dist-test-slave-rxv6"
server_key: "de5f0fb63e02f62e1d0a5324bee7794d"
server_key_iv: "7b7b31eb8efcca8b735f8f897693b42b"
server_key_version: "encryptionkey@0"
I20260619 18:38:11.604523 8803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:11.627122 8803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:11.630214 8803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:11.630383 8803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:11.630945 8803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:11.631739 8803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:11.631795 8803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:11.631848 8803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:11.631875 8803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:11.640215 8803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34503
I20260619 18:38:11.640237 8931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34503 every 8 connection(s)
I20260619 18:38:11.641121 8803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:38:11.647756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 8803
I20260619 18:38:11.647873 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:38:11.648130 392 external_mini_cluster.cc:1428] Setting key f475259c1428dc043720790e94cd5367
Jun 19 18:38:11 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:38:11.655035 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:11.642966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44447 (local address 127.0.98.62:35899)
0619 18:38:11.643104 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:11.643109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:11.644063 (+ 954us) server_negotiation.cc:407] Connection header received
0619 18:38:11.645467 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:11.645471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:11.645530 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:11.645603 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:11.646864 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:11.647388 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:11.648641 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:11.648861 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:11.648959 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:11.651706 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:11.651710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:11.651713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:11.651743 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:11.653037 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:11.653533 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:11.653537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:11.653539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:11.653623 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:11.653959 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:11.653963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:11.653965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:11.654153 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:11.654237 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:11.654744 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 18:38:11.654867 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":49}
I20260619 18:38:11.655840 8934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:11.643376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:44447)
0619 18:38:11.643872 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:11.643920 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:11.645230 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:11.645778 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:11.645786 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:11.646188 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:11.646709 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:11.646724 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:11.647515 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:11.647518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:11.648518 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:11.648528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:11.649005 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:11.649010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:11.649191 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:11.649953 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:11.649975 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:11.651511 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:11.653157 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:11.653163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:11.653181 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:11.653438 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:11.653705 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:11.653709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:11.653711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:11.653857 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:11.654250 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:11.654257 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:11.654508 (+ 251us) client_negotiation.cc:769] Sending connection context
0619 18:38:11.654673 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 18:38:11.655008 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 18:38:11.657039 8932 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35899
I20260619 18:38:11.657279 8932 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:11.657801 8932 heartbeater.cc:511] Master 127.0.98.62:35899 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 18:38:11.658939 8600 ts_manager.cc:194] Registered new tserver with Master: 31a3b574fed342bf88ac3c532d231fb6 (127.0.98.2:34503)
I20260619 18:38:11.659523 8600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:38:11.673406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35899
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:11.781543 8937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:11.781795 8937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:11.781855 8937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:11.785871 8937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:11.785953 8937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:11.786036 8937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:38:11.791265 8937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:11.792264 8937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:11.793267 8937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:11.803498 8942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:11.803486 8943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:11.803704 8937 server_base.cc:1034] running on GCE node
W20260619 18:38:11.803491 8945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:11.804222 8937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:11.804864 8937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:11.805962 8937 hybrid_clock.cc:648] HybridClock initialized: now 1781894291805947 us; error 48 us; skew 500 ppm
Jun 19 18:38:11 dist-test-slave-rxv6 krb5kdc[8562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:11.808766 8937 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:38:11.809620 8937 webserver.cc:460] Webserver started at http://127.0.98.3:34567/ using document root <none> and password file <none>
I20260619 18:38:11.810231 8937 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:11.810299 8937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:11.810499 8937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:11.812505 8937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "7b0ebf7f6d224dc594558e22c1d81352"
format_stamp: "Formatted at 2026-06-19 18:38:11 on dist-test-slave-rxv6"
server_key: "a54a7ecb405d274483412dd9dec09940"
server_key_iv: "c965daf4641d63e4e8c0ba66ac5bb3e5"
server_key_version: "encryptionkey@0"
I20260619 18:38:11.813081 8937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7b0ebf7f6d224dc594558e22c1d81352"
format_stamp: "Formatted at 2026-06-19 18:38:11 on dist-test-slave-rxv6"
server_key: "a54a7ecb405d274483412dd9dec09940"
server_key_iv: "c965daf4641d63e4e8c0ba66ac5bb3e5"
server_key_version: "encryptionkey@0"
I20260619 18:38:11.816789 8937 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 18:38:11.819204 8952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:11.820415 8937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:38:11.820560 8937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "7b0ebf7f6d224dc594558e22c1d81352"
format_stamp: "Formatted at 2026-06-19 18:38:11 on dist-test-slave-rxv6"
server_key: "a54a7ecb405d274483412dd9dec09940"
server_key_iv: "c965daf4641d63e4e8c0ba66ac5bb3e5"
server_key_version: "encryptionkey@0"
I20260619 18:38:11.820672 8937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:11.831174 8937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:11.834295 8937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:11.834471 8937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:11.835027 8937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:11.835819 8937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:11.835873 8937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:11.835924 8937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:11.835951 8937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:11.844233 8937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32961
I20260619 18:38:11.844280 9065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32961 every 8 connection(s)
I20260619 18:38:11.845136 8937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:38:11.849175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 8937
I20260619 18:38:11.849313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:38:11.849596 392 external_mini_cluster.cc:1428] Setting key 8f6054e16a770d6ea96b07f3f4eab36a
Jun 19 18:38:11 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:38:11.858104 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:11.846928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56767 (local address 127.0.98.62:35899)
0619 18:38:11.847063 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:11.847067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:11.847844 (+ 777us) server_negotiation.cc:407] Connection header received
0619 18:38:11.849258 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:11.849262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:11.849317 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:11.849409 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:11.850725 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:11.851440 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:11.852517 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:11.852749 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:11.852825 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:11.855132 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:11.855139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:11.855141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:11.855165 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:11.856453 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:11.856888 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:11.856891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:11.856892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:11.856955 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:11.857211 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:11.857213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:11.857214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:11.857374 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:11.857441 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:11.857858 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 18:38:11.857981 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:38:11.858918 9068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:11.847297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:56767)
0619 18:38:11.847705 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:11.847746 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:11.849002 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:11.849600 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:11.849609 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:11.850067 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:11.850586 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:11.850604 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:11.851540 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:11.851543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:11.852424 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:11.852433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:11.852856 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:11.852859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:11.852990 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:11.853536 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:11.853552 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:11.854967 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:11.856554 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:11.856558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:11.856571 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:11.856816 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:11.857014 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:11.857016 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:11.857017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:11.857131 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:11.857442 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:11.857446 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:11.857671 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 18:38:11.857821 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 18:38:11.858105 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":98,"threads_started":1}
I20260619 18:38:11.859977 9066 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35899
I20260619 18:38:11.860198 9066 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:11.860635 9066 heartbeater.cc:511] Master 127.0.98.62:35899 requested a full tablet report, sending...
I20260619 18:38:11.861594 8600 ts_manager.cc:194] Registered new tserver with Master: 7b0ebf7f6d224dc594558e22c1d81352 (127.0.98.3:32961)
I20260619 18:38:11.862260 8600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56767
I20260619 18:38:11.863199 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 18:38:11.932013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 8669
I20260619 18:38:11.937048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 8803
I20260619 18:38:11.941618 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 8937
I20260619 18:38:11.946064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 8575
I20260619 18:38:11.951336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35899
--webserver_interface=127.0.98.62
--webserver_port=41181
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:12.061575 9075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:12.061856 9075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:12.061926 9075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:12.066020 9075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:38:12.066098 9075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:12.066118 9075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:12.066133 9075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:38:12.066146 9075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:38:12.071915 9075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:12.073030 9075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:12.074054 9075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:12.082541 9081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:12.082541 9080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:12.082541 9083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:12.082775 9075 server_base.cc:1034] running on GCE node
I20260619 18:38:12.083464 9075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:12.084678 9075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:12.085860 9075 hybrid_clock.cc:648] HybridClock initialized: now 1781894292085845 us; error 32 us; skew 500 ppm
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894292, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:12.088841 9075 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:38:12.089864 9075 webserver.cc:460] Webserver started at http://127.0.98.62:41181/ using document root <none> and password file <none>
I20260619 18:38:12.090539 9075 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:12.090637 9075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:12.095211 9075 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 18:38:12.097810 9090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:12.099331 9075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:38:12.099475 9075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "1850f66ec25a4d64b56de89dadb15358"
format_stamp: "Formatted at 2026-06-19 18:38:11 on dist-test-slave-rxv6"
server_key: "317c00aa5703abed2f8a4c2b6d01908a"
server_key_iv: "a2dd3cf805ab0c32be84a3ac6e554491"
server_key_version: "encryptionkey@0"
I20260619 18:38:12.100100 9075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:12.115743 9075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:12.119144 9075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:12.119339 9075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:12.126207 9075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35899
I20260619 18:38:12.126232 9142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35899 every 8 connection(s)
I20260619 18:38:12.127310 9075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:38:12.127633 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 9075
I20260619 18:38:12.128103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40265
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35899
--webserver_port=45077
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:38:12.131841 9143 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:38:12.133585 9143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358: Bootstrap starting.
I20260619 18:38:12.151265 9143 log.cc:824] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:12.158596 9143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:38:12.158969 9143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358: Bootstrap complete.
I20260619 18:38:12.163240 9143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1850f66ec25a4d64b56de89dadb15358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35899 } }
I20260619 18:38:12.163669 9143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1850f66ec25a4d64b56de89dadb15358, State: Initialized, Role: FOLLOWER
I20260619 18:38:12.164145 9143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1850f66ec25a4d64b56de89dadb15358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35899 } }
I20260619 18:38:12.164273 9143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:38:12.164358 9143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:38:12.164484 9143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:38:12.166741 9143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1850f66ec25a4d64b56de89dadb15358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35899 } }
I20260619 18:38:12.167065 9143 leader_election.cc:302] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1850f66ec25a4d64b56de89dadb15358; no voters:
I20260619 18:38:12.167402 9143 leader_election.cc:288] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:38:12.167505 9147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:38:12.167685 9147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [term 2 LEADER]: Becoming Leader. State: Replica: 1850f66ec25a4d64b56de89dadb15358, State: Running, Role: LEADER
I20260619 18:38:12.168036 9147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1850f66ec25a4d64b56de89dadb15358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35899 } }
I20260619 18:38:12.168634 9143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:38:12.169437 9148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1850f66ec25a4d64b56de89dadb15358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1850f66ec25a4d64b56de89dadb15358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35899 } } }
I20260619 18:38:12.169565 9148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:12.169898 9156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:38:12.169899 9149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1850f66ec25a4d64b56de89dadb15358. Latest consensus state: current_term: 2 leader_uuid: "1850f66ec25a4d64b56de89dadb15358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1850f66ec25a4d64b56de89dadb15358" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35899 } } }
I20260619 18:38:12.170054 9149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:12.173151 9156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:38:12.173840 9156 catalog_manager.cc:1259] Loaded cluster ID: 8f74a76e6f2a4d26a4f148c18636658a
I20260619 18:38:12.173895 9156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:38:12.175585 9156 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:38:12.176119 9156 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1850f66ec25a4d64b56de89dadb15358: Loaded TSK: 0
I20260619 18:38:12.177031 9156 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:38:12.249354 9145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:12.249660 9145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:12.249722 9145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:12.253808 9145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:12.253890 9145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:12.253997 9145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:38:12.259461 9145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45077
--tserver_master_addrs=127.0.98.62:35899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:12.260591 9145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:12.261698 9145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:12.272219 9169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:12.272229 9170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:12.272531 9145 server_base.cc:1034] running on GCE node
W20260619 18:38:12.272678 9172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:12.273058 9145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:12.273747 9145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:12.274940 9145 hybrid_clock.cc:648] HybridClock initialized: now 1781894292274905 us; error 49 us; skew 500 ppm
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894292, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:12.277668 9145 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 18:38:12.278645 9145 webserver.cc:460] Webserver started at http://127.0.98.1:45077/ using document root <none> and password file <none>
I20260619 18:38:12.279259 9145 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:12.279335 9145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:12.283238 9145 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:38:12.285609 9179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:12.287170 9145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:38:12.287298 9145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "076732515a434d03be9469d49448f9c7"
format_stamp: "Formatted at 2026-06-19 18:38:11 on dist-test-slave-rxv6"
server_key: "62918d9d2cd85643902d318f14b150b7"
server_key_iv: "41abf153cc2fb2ff54e4ac2d3fb22f00"
server_key_version: "encryptionkey@0"
I20260619 18:38:12.287933 9145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:12.303402 9145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:12.306875 9145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:12.307065 9145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:12.307677 9145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:12.308531 9145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:12.308579 9145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:12.308652 9145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:12.308677 9145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:12.317394 9145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40265
I20260619 18:38:12.317418 9292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40265 every 8 connection(s)
I20260619 18:38:12.318427 9145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:38:12.324559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 9145
I20260619 18:38:12.324965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34503
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35899
--webserver_port=38013
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:38:12.331557 9295 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:12.320478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35531 (local address 127.0.98.62:35899)
0619 18:38:12.320977 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:12.320997 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:12.321181 (+ 184us) server_negotiation.cc:407] Connection header received
0619 18:38:12.322786 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:12.322798 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:12.323089 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:12.323349 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:12.324506 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.325548 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.326704 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.327170 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.327325 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.330886 (+ 3561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35531: BlockingRecv error: recv got EOF from 127.0.98.1:35531 (error 108)
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":111,"threads_started":1}
I20260619 18:38:12.331631 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:12.320582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:35531)
0619 18:38:12.321035 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:12.321076 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:12.322383 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:12.323435 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:12.323442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:12.323846 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:12.324340 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.324354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.325677 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.325679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.326575 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.326594 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.327321 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.327325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.327493 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.328117 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:12.328134 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:12.330623 (+ 2489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":131,"threads_started":1}
W20260619 18:38:12.332239 9293 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35899 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35899: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:12.337694 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:12.332718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:56755)
0619 18:38:12.332846 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:12.332860 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:12.332948 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:12.333178 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:12.333181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:12.333337 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:12.333430 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.333436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.334209 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.334211 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.334759 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.334767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.335168 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.335172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.335254 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.335728 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:12.335738 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:12.337562 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 18:38:12.337867 9295 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:12.332789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56755 (local address 127.0.98.62:35899)
0619 18:38:12.332893 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:12.332896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:12.332910 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:38:12.333034 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:12.333038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:12.333083 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:12.333154 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:12.333536 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.334106 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.334864 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.335074 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.335160 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.337767 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56755: BlockingRecv error: recv got EOF from 127.0.98.1:56755 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:12.343670 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:12.338414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:38835)
0619 18:38:12.338511 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:12.338524 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:12.338640 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:12.338865 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:12.338867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:12.338984 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:12.339054 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.339058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.340020 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.340023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.340546 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.340554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.341146 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.341148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.341208 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.341688 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:12.341700 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:12.343553 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 18:38:12.343868 9293 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:38:12.344031 9295 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:12.338501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38835 (local address 127.0.98.62:35899)
0619 18:38:12.338614 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:12.338618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:12.338632 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:38:12.338721 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:12.338724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:12.338776 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:12.338856 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:12.339137 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.339919 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.340685 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.341037 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.341134 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.343705 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38835: BlockingRecv error: recv got EOF from 127.0.98.1:38835 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 18:38:12.435899 9297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:12.436175 9297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:12.436218 9297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:12.440186 9297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:12.440254 9297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:12.440315 9297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:38:12.445706 9297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34503
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38013
--tserver_master_addrs=127.0.98.62:35899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:12.446892 9297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:12.447903 9297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:12.457542 9303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:12.457595 9302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:12.457598 9305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:12.457947 9297 server_base.cc:1034] running on GCE node
I20260619 18:38:12.458328 9297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:12.458968 9297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:12.460125 9297 hybrid_clock.cc:648] HybridClock initialized: now 1781894292460112 us; error 36 us; skew 500 ppm
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894292, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:12.462996 9297 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 18:38:12.463935 9297 webserver.cc:460] Webserver started at http://127.0.98.2:38013/ using document root <none> and password file <none>
I20260619 18:38:12.464489 9297 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:12.464567 9297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:12.468873 9297 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:38:12.471233 9312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:12.472576 9297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:38:12.472734 9297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "31a3b574fed342bf88ac3c532d231fb6"
format_stamp: "Formatted at 2026-06-19 18:38:11 on dist-test-slave-rxv6"
server_key: "de5f0fb63e02f62e1d0a5324bee7794d"
server_key_iv: "7b7b31eb8efcca8b735f8f897693b42b"
server_key_version: "encryptionkey@0"
I20260619 18:38:12.473315 9297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:12.500264 9297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:12.503593 9297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:12.503779 9297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:12.504451 9297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:12.505357 9297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:12.505404 9297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:12.505446 9297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:12.505475 9297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:12.514000 9297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34503
I20260619 18:38:12.514025 9425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34503 every 8 connection(s)
I20260619 18:38:12.515040 9297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:38:12.521154 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 9297
I20260619 18:38:12.521592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:32961
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35899
--webserver_port=34567
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:38:12.526788 9295 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:12.516995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37883 (local address 127.0.98.62:35899)
0619 18:38:12.517127 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:12.517131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:12.517984 (+ 853us) server_negotiation.cc:407] Connection header received
0619 18:38:12.519382 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:12.519386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:12.519448 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:12.519522 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:12.520747 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.521269 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.522510 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.522866 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.522957 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.526638 (+ 3681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37883: BlockingRecv error: recv got EOF from 127.0.98.2:37883 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 18:38:12.527300 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:12.517381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:37883)
0619 18:38:12.517820 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:12.517863 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:12.519165 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:12.519720 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:12.519728 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:12.520098 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:12.520591 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.520604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.521405 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.521407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.522392 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.522402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.523032 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.523036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.523267 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.523891 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:12.523909 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:12.526451 (+ 2542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":102,"threads_started":1}
W20260619 18:38:12.527907 9426 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35899 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35899: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:12.533722 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:12.528361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:56893)
0619 18:38:12.528591 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:12.528603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:12.528726 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:12.528966 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:12.528968 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:12.529095 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:12.529189 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.529193 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.530216 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.530219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.530794 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.530802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.531358 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.531361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.531433 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.531882 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:12.531893 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:12.533619 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166}
W20260619 18:38:12.534143 9295 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:12.528477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56893 (local address 127.0.98.62:35899)
0619 18:38:12.528603 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:12.528607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:12.528644 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:38:12.528814 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:12.528817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:12.528869 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:12.528958 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:12.529287 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.530110 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.530904 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.531236 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.531339 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.533790 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56893: BlockingRecv error: recv got EOF from 127.0.98.2:56893 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:12.539253 9428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:12.534220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:48923)
0619 18:38:12.534403 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:12.534418 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:12.534485 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:12.534703 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:12.534705 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:12.534835 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:12.534930 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.534935 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.535885 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.535887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.536399 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.536407 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.536906 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.536908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.536977 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.537438 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:12.537449 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:12.539155 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114}
W20260619 18:38:12.539417 9295 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:12.534331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48923 (local address 127.0.98.62:35899)
0619 18:38:12.534437 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:12.534441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:12.534456 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:38:12.534563 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:12.534566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:12.534608 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:12.534679 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:12.535014 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.535795 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.536485 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.536819 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.536908 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.539305 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48923: BlockingRecv error: recv got EOF from 127.0.98.2:48923 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 18:38:12.539582 9426 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:38:12.635986 9429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:12.636281 9429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:12.636325 9429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:12.640394 9429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:12.640475 9429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:12.640537 9429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:38:12.646013 9429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:32961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34567
--tserver_master_addrs=127.0.98.62:35899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:12.647094 9429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:12.648151 9429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:12.657845 9434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:12.657881 9429 server_base.cc:1034] running on GCE node
W20260619 18:38:12.657923 9437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:12.657887 9435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:12.658556 9429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:12.659282 9429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:12.660454 9429 hybrid_clock.cc:648] HybridClock initialized: now 1781894292660427 us; error 39 us; skew 500 ppm
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894292, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:12.663455 9429 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 18:38:12.664496 9429 webserver.cc:460] Webserver started at http://127.0.98.3:34567/ using document root <none> and password file <none>
I20260619 18:38:12.665071 9429 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:12.665143 9429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:12.669473 9429 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:38:12.671877 9444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:12.673278 9429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:38:12.673400 9429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "7b0ebf7f6d224dc594558e22c1d81352"
format_stamp: "Formatted at 2026-06-19 18:38:11 on dist-test-slave-rxv6"
server_key: "a54a7ecb405d274483412dd9dec09940"
server_key_iv: "c965daf4641d63e4e8c0ba66ac5bb3e5"
server_key_version: "encryptionkey@0"
I20260619 18:38:12.674012 9429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:12.685702 9429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:12.688824 9429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:12.688997 9429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:12.689579 9429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:12.690475 9429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:12.690524 9429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:12.690577 9429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:12.690604 9429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:12.699028 9429 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32961
I20260619 18:38:12.699059 9557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32961 every 8 connection(s)
I20260619 18:38:12.700105 9429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:38:12.708485 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 9429
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:38:12.711225 9295 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:12.702011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49577 (local address 127.0.98.62:35899)
0619 18:38:12.702155 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:12.702159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:12.703025 (+ 866us) server_negotiation.cc:407] Connection header received
0619 18:38:12.704470 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:12.704473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:12.704530 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:12.704600 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:12.705784 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.706346 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.707459 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.707677 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.707765 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.711093 (+ 3328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49577: BlockingRecv error: recv got EOF from 127.0.98.3:49577 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:38:12.711758 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:12.702397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:49577)
0619 18:38:12.702870 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:12.702913 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:12.704237 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:12.704794 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:12.704801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:12.705155 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:12.705648 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.705662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.706456 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.706460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.707347 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.707356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.707810 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.707814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.707978 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.708551 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:12.708572 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:12.710971 (+ 2399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":101,"threads_started":1}
W20260619 18:38:12.712226 9558 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35899 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35899: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:12.717187 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:12.712647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:35227)
0619 18:38:12.712745 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:12.712759 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:12.712825 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:12.713006 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:12.713008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:12.713164 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:12.713254 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.713258 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.713997 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.714000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.714616 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.714624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.715053 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.715057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.715134 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.715440 (+ 306us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:12.715453 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:12.717081 (+ 1628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 18:38:12.717479 9295 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:12.712731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35227 (local address 127.0.98.62:35899)
0619 18:38:12.712835 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:12.712839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:12.712850 (+ 11us) server_negotiation.cc:407] Connection header received
0619 18:38:12.712882 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:12.712885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:12.712921 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:12.712994 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:12.713357 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.713862 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.714709 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.714955 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.715029 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.717218 (+ 2189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35227: BlockingRecv error: recv got EOF from 127.0.98.3:35227 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 18:38:12.718940 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:12.709598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.0.98.62:35899)
0619 18:38:12.709848 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:12.709851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:12.709861 (+ 10us) server_negotiation.cc:407] Connection header received
0619 18:38:12.709887 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:12.709889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:12.709947 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:12.710024 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:12.710726 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.711384 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.712387 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.712684 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.712778 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.714323 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:12.714328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:12.714342 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:12.714369 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:12.716172 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:12.716778 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:12.716784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:12.716792 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:12.716938 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:12.717233 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:12.717237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:12.717238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:12.717698 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:12.717877 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:12.718642 (+ 765us) server_negotiation.cc:299] Negotiation successful
0619 18:38:12.718832 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":97,"threads_started":1}
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:12 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:12.722502 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:12.717806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:46421)
0619 18:38:12.717928 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:12.717942 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:12.718008 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:12.718186 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:12.718190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:12.718330 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:12.718413 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.718420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.719130 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.719132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.719630 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:12.719637 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.720161 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.720165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:12.720240 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.720659 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:12.720671 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:12.722415 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 18:38:12.722596 9295 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:12.717886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46421 (local address 127.0.98.62:35899)
0619 18:38:12.718023 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:12.718025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:12.718038 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:38:12.718066 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:12.718068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:12.718098 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:12.718155 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:12.718518 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.719039 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.719726 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:12.720024 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:12.720103 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:12.722514 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46421: BlockingRecv error: recv got EOF from 127.0.98.3:46421 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 18:38:12.722683 9558 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 18:38:13 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:13 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:13.349756 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:13.344553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:55185)
0619 18:38:13.344884 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:13.344900 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:13.345017 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:13.345324 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:13.345326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:13.345495 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:13.345601 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:13.345606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:13.346363 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:13.346365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:13.346964 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:13.346973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:13.347381 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:13.347384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:13.347452 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:13.347977 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:13.347988 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:13.349631 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":87,"threads_started":1}
W20260619 18:38:13.349871 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:13.344620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55185 (local address 127.0.98.62:35899)
0619 18:38:13.344892 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:13.344896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:13.344958 (+ 62us) server_negotiation.cc:407] Connection header received
0619 18:38:13.345104 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:13.345107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:13.345159 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:13.345249 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:13.345740 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:13.346276 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:13.347067 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:13.347292 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:13.347363 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:13.349760 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55185: BlockingRecv error: recv got EOF from 127.0.98.1:55185 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":90,"threads_started":1}
Jun 19 18:38:13 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:13 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:13.545507 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:13.540275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:34513)
0619 18:38:13.540569 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:13.540584 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:13.540701 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:13.540953 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:13.540956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:13.541149 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:13.541256 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:13.541262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:13.542033 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:13.542036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:13.542626 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:13.542635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:13.543060 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:13.543064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:13.543153 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:13.543629 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:13.543640 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:13.545381 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":86,"threads_started":1}
W20260619 18:38:13.545727 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:13.540385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34513 (local address 127.0.98.62:35899)
0619 18:38:13.540548 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:13.540552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:13.540657 (+ 105us) server_negotiation.cc:407] Connection header received
0619 18:38:13.540791 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:13.540795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:13.540845 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:13.540920 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:13.541390 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:13.541944 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:13.542740 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:13.542962 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:13.543044 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:13.545520 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34513: BlockingRecv error: recv got EOF from 127.0.98.2:34513 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 18:38:13 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:13 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:13.730053 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:13.723466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:40423)
0619 18:38:13.723808 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:13.723824 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:13.723983 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:13.724370 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:13.724373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:13.724604 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:13.724775 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:13.724782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:13.725644 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:13.725648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:13.726411 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:13.726424 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:13.726971 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:13.726975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:13.727087 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:13.727633 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:13.727646 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:13.729840 (+ 2194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":81,"threads_started":1}
W20260619 18:38:13.730216 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:13.723638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40423 (local address 127.0.98.62:35899)
0619 18:38:13.723841 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:13.723845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:13.723941 (+ 96us) server_negotiation.cc:407] Connection header received
0619 18:38:13.724159 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:13.724161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:13.724218 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:13.724310 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:13.724932 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:13.725497 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:13.726566 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:13.726840 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:13.726946 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:13.730073 (+ 3127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40423: BlockingRecv error: recv got EOF from 127.0.98.3:40423 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Jun 19 18:38:14 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:14 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:14.360874 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:14.350645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:48591)
0619 18:38:14.351012 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:14.351026 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:14.351147 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:14.351393 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:14.351395 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:14.351597 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:14.351722 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:14.351728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:14.352531 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:14.352534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:14.353174 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:14.353183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:14.353584 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:14.353586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:14.353647 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:14.358738 (+ 5091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:14.358752 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:14.360675 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":106,"threads_started":1}
W20260619 18:38:14.361052 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:14.350695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48591 (local address 127.0.98.62:35899)
0619 18:38:14.351011 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:14.351015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:14.351101 (+ 86us) server_negotiation.cc:407] Connection header received
0619 18:38:14.351225 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:14.351227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:14.351272 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:14.351361 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:14.351836 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:14.352413 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:14.353271 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:14.353494 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:14.353570 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:14.360838 (+ 7268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48591: BlockingRecv error: recv got EOF from 127.0.98.1:48591 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
Jun 19 18:38:14 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:14 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:14.551905 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:14.546362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:49683)
0619 18:38:14.546643 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:14.546659 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:14.546777 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:14.547055 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:14.547057 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:14.547230 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:14.547327 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:14.547332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:14.548100 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:14.548102 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:14.548660 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:14.548667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:14.549058 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:14.549060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:14.549119 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:14.549616 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:14.549628 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:14.551724 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
W20260619 18:38:14.552035 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:14.546480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49683 (local address 127.0.98.62:35899)
0619 18:38:14.546656 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:14.546660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:14.546740 (+ 80us) server_negotiation.cc:407] Connection header received
0619 18:38:14.546869 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:14.546871 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:14.546929 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:14.547028 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:14.547433 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:14.548013 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:14.548755 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:14.548975 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:14.549048 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:14.551892 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49683: BlockingRecv error: recv got EOF from 127.0.98.2:49683 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 18:38:14 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:14 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:14.736680 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:14.730899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:34101)
0619 18:38:14.731184 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:14.731198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:14.731312 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:14.731527 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:14.731529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:14.731735 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:14.731866 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:14.731871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:14.732712 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:14.732714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:14.733291 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:14.733298 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:14.733747 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:14.733749 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:14.733828 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:14.734412 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:14.734426 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:14.736516 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
W20260619 18:38:14.736809 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:14.731039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34101 (local address 127.0.98.62:35899)
0619 18:38:14.731204 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:14.731208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:14.731266 (+ 58us) server_negotiation.cc:407] Connection header received
0619 18:38:14.731384 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:14.731387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:14.731437 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:14.731502 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:14.732023 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:14.732626 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:14.733401 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:14.733649 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:14.733729 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:14.736681 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34101: BlockingRecv error: recv got EOF from 127.0.98.3:34101 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 19 18:38:15 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:15 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:15.367383 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:15.361738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:55987)
0619 18:38:15.362069 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:15.362085 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:15.362196 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:15.362434 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:15.362436 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:15.362639 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:15.362745 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:15.362750 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:15.363600 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:15.363603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:15.364219 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:15.364229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:15.364652 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:15.364654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:15.364717 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:15.365279 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:15.365291 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:15.367236 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":88,"threads_started":1}
W20260619 18:38:15.367525 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:15.361823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55987 (local address 127.0.98.62:35899)
0619 18:38:15.362208 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:15.362212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:15.362248 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:38:15.362285 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:15.362288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:15.362334 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:15.362407 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:15.362869 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:15.363499 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:15.364329 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:15.364561 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:15.364641 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:15.367385 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55987: BlockingRecv error: recv got EOF from 127.0.98.1:55987 (error 108)
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":148,"threads_started":1}
Jun 19 18:38:15 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:15 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:15.558884 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:15.552860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:50251)
0619 18:38:15.553155 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:15.553170 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:15.553330 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:15.553603 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:15.553606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:15.553816 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:15.553980 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:15.553987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:15.554785 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:15.554789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:15.555550 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:15.555560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:15.556078 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:15.556082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:15.556163 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:15.556739 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:15.556751 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:15.558718 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":88,"threads_started":1}
W20260619 18:38:15.558981 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:15.552946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50251 (local address 127.0.98.62:35899)
0619 18:38:15.553107 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:15.553112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:15.553237 (+ 125us) server_negotiation.cc:407] Connection header received
0619 18:38:15.553412 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:15.553414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:15.553464 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:15.553532 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:15.554085 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:15.554680 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:15.555660 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:15.555962 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:15.556052 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:15.558869 (+ 2817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50251: BlockingRecv error: recv got EOF from 127.0.98.2:50251 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 18:38:15 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:15 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:15.743629 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:15.737538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:41261)
0619 18:38:15.737861 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:15.737875 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:15.738030 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:15.738262 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:15.738264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:15.738478 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:15.738591 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:15.738597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:15.739533 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:15.739536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:15.740122 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:15.740130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:15.740545 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:15.740547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:15.740618 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:15.741140 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:15.741152 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:15.743456 (+ 2304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":85,"threads_started":1}
W20260619 18:38:15.743753 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:15.737660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41261 (local address 127.0.98.62:35899)
0619 18:38:15.737839 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:15.737843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:15.738002 (+ 159us) server_negotiation.cc:407] Connection header received
0619 18:38:15.738110 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:15.738112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:15.738167 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:15.738235 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:15.738714 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:15.739372 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:15.740220 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:15.740460 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:15.740533 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:15.743635 (+ 3102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41261: BlockingRecv error: recv got EOF from 127.0.98.3:41261 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 19 18:38:16 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:16 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:16.373883 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:16.368208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:37473)
0619 18:38:16.368521 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:16.368536 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:16.368684 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:16.368958 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:16.368960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:16.369174 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:16.369291 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:16.369297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:16.370179 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:16.370183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:16.370840 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:16.370851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:16.371283 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:16.371286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:16.371352 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:16.371877 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:16.371889 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:16.373744 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":108,"threads_started":1}
W20260619 18:38:16.374065 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:16.368288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37473 (local address 127.0.98.62:35899)
0619 18:38:16.368546 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:16.368550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:16.368651 (+ 101us) server_negotiation.cc:407] Connection header received
0619 18:38:16.368783 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:16.368786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:16.368838 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:16.368926 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:16.369378 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:16.370086 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:16.370950 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:16.371184 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:16.371262 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:16.373898 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37473: BlockingRecv error: recv got EOF from 127.0.98.1:37473 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":69,"threads_started":1}
Jun 19 18:38:16 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:16 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:16.565820 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:16.559779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:54481)
0619 18:38:16.560098 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:16.560111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:16.560228 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:16.560488 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:16.560491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:16.560695 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:16.560832 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:16.560838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:16.561680 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:16.561683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:16.562351 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:16.562360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:16.562787 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:16.562790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:16.562858 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:16.563448 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:16.563495 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:16.565644 (+ 2149us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":113,"threads_started":1}
W20260619 18:38:16.565953 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:16.559879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54481 (local address 127.0.98.62:35899)
0619 18:38:16.560085 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:16.560089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:16.560171 (+ 82us) server_negotiation.cc:407] Connection header received
0619 18:38:16.560298 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:16.560300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:16.560349 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:16.560418 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:16.560964 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:16.561561 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:16.562454 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:16.562696 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:16.562773 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:16.565810 (+ 3037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54481: BlockingRecv error: recv got EOF from 127.0.98.2:54481 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 18:38:16 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:16 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:16.755076 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:16.744446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:53613)
0619 18:38:16.744761 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:16.744774 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:16.744892 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:16.745174 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:16.745177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:16.745387 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:16.745508 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:16.745514 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:16.746344 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:16.746346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:16.746945 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:16.746954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:16.747353 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:16.747355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:16.747416 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:16.752787 (+ 5371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:16.752802 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:16.754920 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":114,"threads_started":1}
W20260619 18:38:16.755167 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:16.744518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53613 (local address 127.0.98.62:35899)
0619 18:38:16.744687 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:16.744691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:16.744849 (+ 158us) server_negotiation.cc:407] Connection header received
0619 18:38:16.744981 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:16.744984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:16.745040 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:16.745111 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:16.745629 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:16.746266 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:16.747043 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:16.747269 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:16.747344 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:16.755061 (+ 7717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53613: BlockingRecv error: recv got EOF from 127.0.98.3:53613 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 18:38:17 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:17 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:17.380685 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:17.374779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:34787)
0619 18:38:17.375097 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:17.375111 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:17.375221 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:17.375447 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:17.375448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:17.375652 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:17.375762 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:17.375767 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:17.376559 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:17.376562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:17.377211 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:17.377222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:17.377716 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:17.377720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:17.377791 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:17.378347 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:17.378361 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:17.380500 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":109,"threads_started":1}
W20260619 18:38:17.380795 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:17.374837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34787 (local address 127.0.98.62:35899)
0619 18:38:17.375108 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:17.375112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:17.375168 (+ 56us) server_negotiation.cc:407] Connection header received
0619 18:38:17.375295 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:17.375298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:17.375346 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:17.375423 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:17.375847 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:17.376472 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:17.377313 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:17.377607 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:17.377710 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:17.380669 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34787: BlockingRecv error: recv got EOF from 127.0.98.1:34787 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
Jun 19 18:38:17 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:17 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:17.572968 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:17.566829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:57841)
0619 18:38:17.567149 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:17.567164 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:17.567272 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:17.567504 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:17.567506 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:17.567728 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:17.567839 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:17.567844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:17.568720 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:17.568724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:17.569555 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:17.569567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:17.570059 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:17.570062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:17.570141 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:17.570706 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:17.570719 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:17.572756 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":113,"threads_started":1}
W20260619 18:38:17.573052 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:17.566983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57841 (local address 127.0.98.62:35899)
0619 18:38:17.567137 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:17.567141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:17.567229 (+ 88us) server_negotiation.cc:407] Connection header received
0619 18:38:17.567365 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:17.567368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:17.567417 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:17.567479 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:17.567926 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:17.568590 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:17.569678 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:17.569934 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:17.570030 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:17.572928 (+ 2898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57841: BlockingRecv error: recv got EOF from 127.0.98.2:57841 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 18:38:17 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:17 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:17.761539 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:17.755892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:50591)
0619 18:38:17.756202 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:17.756217 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:17.756330 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:17.756559 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:17.756561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:17.756765 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:17.756882 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:17.756887 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:17.757730 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:17.757733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:17.758423 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:17.758433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:17.758866 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:17.758868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:17.758930 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:17.759430 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:17.759442 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:17.761352 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":95,"threads_started":1}
W20260619 18:38:17.761600 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:17.755987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50591 (local address 127.0.98.62:35899)
0619 18:38:17.756142 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:17.756147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:17.756278 (+ 131us) server_negotiation.cc:407] Connection header received
0619 18:38:17.756411 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:17.756413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:17.756462 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:17.756519 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:17.757004 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:17.757610 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:17.758536 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:17.758771 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:17.758863 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:17.761490 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50591: BlockingRecv error: recv got EOF from 127.0.98.3:50591 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 18:38:18 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:18 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:18.387813 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:18.381782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:42573)
0619 18:38:18.382149 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:18.382168 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:18.382307 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:18.382680 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:18.382684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:18.382904 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:18.383055 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:18.383065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:18.383855 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:18.383859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:18.384449 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:18.384458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:18.384902 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:18.384905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:18.384962 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:18.385519 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:18.385533 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:18.387676 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":129,"threads_started":1}
W20260619 18:38:18.387950 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:18.381867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42573 (local address 127.0.98.62:35899)
0619 18:38:18.382205 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:18.382210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:18.382227 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:38:18.382419 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:18.382422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:18.382485 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:18.382585 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:18.383164 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:18.383745 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:18.384560 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:18.384812 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:18.384893 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:18.387824 (+ 2931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42573: BlockingRecv error: recv got EOF from 127.0.98.1:42573 (error 108)
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":131,"threads_started":1}
Jun 19 18:38:18 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:18 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:18.579792 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:18.573981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:39229)
0619 18:38:18.574293 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:18.574307 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:18.574426 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:18.574657 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:18.574659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:18.574863 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:18.574971 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:18.574977 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:18.575749 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:18.575754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:18.576353 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:18.576363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:18.576794 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:18.576797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:18.576864 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:18.577449 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:18.577461 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:18.579598 (+ 2137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":122,"threads_started":1}
W20260619 18:38:18.579919 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:18.574009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39229 (local address 127.0.98.62:35899)
0619 18:38:18.574179 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:18.574183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:18.574380 (+ 197us) server_negotiation.cc:407] Connection header received
0619 18:38:18.574504 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:18.574512 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:18.574563 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:18.574633 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:18.575060 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:18.575638 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:18.576451 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:18.576691 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:18.576770 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:18.579800 (+ 3030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39229: BlockingRecv error: recv got EOF from 127.0.98.2:39229 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 18:38:18 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:18 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:18.768078 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:18.762481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:38425)
0619 18:38:18.762829 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:18.762843 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:18.762957 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:18.763166 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:18.763168 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:18.763382 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:18.763497 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:18.763502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:18.764219 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:18.764222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:18.764816 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:18.764824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:18.765194 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:18.765196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:18.765254 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:18.765879 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:18.765891 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:18.767930 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
W20260619 18:38:18.768246 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:18.762559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38425 (local address 127.0.98.62:35899)
0619 18:38:18.762715 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:18.762720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:18.762898 (+ 178us) server_negotiation.cc:407] Connection header received
0619 18:38:18.763023 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:18.763025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:18.763075 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:18.763142 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:18.763582 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:18.764137 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:18.764899 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:18.765114 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:18.765183 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:18.768114 (+ 2931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38425: BlockingRecv error: recv got EOF from 127.0.98.3:38425 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 18:38:19 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:19 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:19.394888 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:19.388719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:46509)
0619 18:38:19.389074 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:19.389090 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:19.389212 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:19.389510 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:19.389512 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:19.389718 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:19.389840 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:19.389846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:19.390784 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:19.390787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:19.391650 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:19.391664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:19.392158 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:19.392163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:19.392255 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:19.392837 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:19.392850 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:19.394752 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":88,"threads_started":1}
W20260619 18:38:19.395071 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:19.388832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46509 (local address 127.0.98.62:35899)
0619 18:38:19.389116 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:19.389120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:19.389152 (+ 32us) server_negotiation.cc:407] Connection header received
0619 18:38:19.389331 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:19.389334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:19.389392 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:19.389479 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:19.389979 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:19.390666 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:19.391792 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:19.392026 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:19.392098 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:19.394923 (+ 2825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46509: BlockingRecv error: recv got EOF from 127.0.98.1:46509 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":89,"threads_started":1}
Jun 19 18:38:19 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:19 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:19.587141 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:19.580710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:60929)
0619 18:38:19.581064 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:19.581080 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:19.581226 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:19.581603 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:19.581606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:19.581819 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:19.582020 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:19.582028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:19.582939 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:19.582943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:19.583676 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:19.583686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:19.584249 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:19.584253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:19.584348 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:19.584955 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:19.584971 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:19.586996 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":136,"threads_started":1}
W20260619 18:38:19.587342 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:19.580829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60929 (local address 127.0.98.62:35899)
0619 18:38:19.581033 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:19.581039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:19.581197 (+ 158us) server_negotiation.cc:407] Connection header received
0619 18:38:19.581319 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:19.581321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:19.581383 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:19.581463 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:19.582179 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:19.582777 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:19.583810 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:19.584135 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:19.584230 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:19.587189 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60929: BlockingRecv error: recv got EOF from 127.0.98.2:60929 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 19 18:38:19 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:19 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:19.774923 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:19.768950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:41989)
0619 18:38:19.769292 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:19.769307 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:19.769469 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:19.769753 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:19.769755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:19.770019 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:19.770137 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:19.770143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:19.770988 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:19.770990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:19.771605 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:19.771612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:19.772045 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:19.772049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:19.772133 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:19.772667 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:19.772679 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:19.774745 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":95,"threads_started":1}
W20260619 18:38:19.775044 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:19.769030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41989 (local address 127.0.98.62:35899)
0619 18:38:19.769196 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:19.769200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:19.769427 (+ 227us) server_negotiation.cc:407] Connection header received
0619 18:38:19.769574 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:19.769577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:19.769644 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:19.769723 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:19.770238 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:19.770901 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:19.771711 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:19.771946 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:19.772033 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:19.774935 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41989: BlockingRecv error: recv got EOF from 127.0.98.3:41989 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 18:38:20 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:20 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:20.401962 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:20.395744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:42173)
0619 18:38:20.396086 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:20.396107 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:20.396249 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:20.396617 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:20.396619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:20.396876 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:20.396994 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:20.397001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:20.397823 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:20.397827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:20.398482 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:20.398492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:20.398969 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:20.398973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:20.399044 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:20.399595 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:20.399606 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:20.401786 (+ 2180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":108,"threads_started":1}
W20260619 18:38:20.402184 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:20.395916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42173 (local address 127.0.98.62:35899)
0619 18:38:20.396184 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:20.396188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:20.396200 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:38:20.396381 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:20.396384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:20.396458 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:20.396545 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:20.397098 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:20.397692 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:20.398598 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:20.398839 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:20.398919 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:20.402029 (+ 3110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42173: BlockingRecv error: recv got EOF from 127.0.98.1:42173 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
Jun 19 18:38:20 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:20 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:20.593722 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:20.587966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:40477)
0619 18:38:20.588276 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:20.588301 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:20.588433 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:20.588689 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:20.588692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:20.588930 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:20.589041 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:20.589049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:20.590002 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:20.590007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:20.590594 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:20.590602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:20.591033 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:20.591036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:20.591117 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:20.591686 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:20.591698 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:20.593584 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":101,"threads_started":1}
W20260619 18:38:20.593842 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:20.588063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40477 (local address 127.0.98.62:35899)
0619 18:38:20.588245 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:20.588250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:20.588383 (+ 133us) server_negotiation.cc:407] Connection header received
0619 18:38:20.588524 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:20.588527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:20.588583 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:20.588653 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:20.589163 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:20.589873 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:20.590702 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:20.590935 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:20.591020 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:20.593733 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40477: BlockingRecv error: recv got EOF from 127.0.98.2:40477 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 18:38:20 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:20 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:20.782051 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:20.775916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:55543)
0619 18:38:20.776264 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:20.776279 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:20.776411 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:20.776698 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:20.776701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:20.776942 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:20.777117 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:20.777127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:20.778102 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:20.778105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:20.778765 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:20.778773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:20.779215 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:20.779217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:20.779277 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:20.779847 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:20.779859 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:20.781848 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":110,"threads_started":1}
W20260619 18:38:20.782182 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:20.775985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55543 (local address 127.0.98.62:35899)
0619 18:38:20.776142 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:20.776148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:20.776361 (+ 213us) server_negotiation.cc:407] Connection header received
0619 18:38:20.776506 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:20.776508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:20.776566 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:20.776652 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:20.777274 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:20.777991 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:20.778866 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:20.779125 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:20.779208 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:20.782068 (+ 2860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55543: BlockingRecv error: recv got EOF from 127.0.98.3:55543 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 18:38:21 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:21 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:21.409328 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:21.402927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:59973)
0619 18:38:21.403334 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:21.403356 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:21.403489 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:21.403785 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:21.403788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:21.404074 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:21.404184 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:21.404191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:21.404993 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:21.404997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:21.405710 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:21.405720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:21.406215 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:21.406219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:21.406296 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:21.406870 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:21.406884 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:21.409160 (+ 2276us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":104,"threads_started":1}
W20260619 18:38:21.409518 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:21.403014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59973 (local address 127.0.98.62:35899)
0619 18:38:21.403329 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:21.403333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:21.403447 (+ 114us) server_negotiation.cc:407] Connection header received
0619 18:38:21.403609 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:21.403612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:21.403669 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:21.403746 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:21.404300 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:21.404870 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:21.405818 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:21.406100 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:21.406185 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:21.409329 (+ 3144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59973: BlockingRecv error: recv got EOF from 127.0.98.1:59973 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":98,"threads_started":1}
Jun 19 18:38:21 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:21 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:21.601033 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:21.594667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:42365)
0619 18:38:21.595084 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:21.595105 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:21.595255 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:21.595551 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:21.595553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:21.595763 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:21.595919 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:21.595926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:21.596798 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:21.596802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:21.597451 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:21.597467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:21.598019 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:21.598023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:21.598105 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:21.598762 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:21.598775 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:21.600864 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":124,"threads_started":1}
W20260619 18:38:21.601177 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:21.594840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42365 (local address 127.0.98.62:35899)
0619 18:38:21.595040 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:21.595046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:21.595204 (+ 158us) server_negotiation.cc:407] Connection header received
0619 18:38:21.595351 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:21.595353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:21.595418 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:21.595489 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:21.596037 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:21.596654 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:21.597607 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:21.597850 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:21.597959 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:21.601025 (+ 3066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42365: BlockingRecv error: recv got EOF from 127.0.98.2:42365 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 18:38:21 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:21 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 18:38:21.797128 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:21.783237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36993 (local address 127.0.98.62:35899)
0619 18:38:21.783410 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:21.783416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:21.783637 (+ 221us) server_negotiation.cc:407] Connection header received
0619 18:38:21.783775 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:21.783778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:21.783840 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:21.783924 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:21.784480 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:21.785262 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:21.786273 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:21.786517 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:21.786604 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:21.796974 (+ 10370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36993: BlockingRecv error: recv got EOF from 127.0.98.3:36993 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:38:21.797501 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:21.783115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:36993)
0619 18:38:21.783495 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:21.783520 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:21.783683 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:21.783982 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:21.783986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:21.784224 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:21.784344 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:21.784351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:21.785387 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:21.785390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:21.786143 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:21.786153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:21.786617 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:21.786620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:21.786700 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:21.793734 (+ 7034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:21.793752 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:21.797319 (+ 3567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":117,"threads_started":1}
Jun 19 18:38:22 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:22 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:22.416260 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:22.410335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:46077)
0619 18:38:22.410640 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:22.410659 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:22.410785 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:22.411064 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:22.411067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:22.411301 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:22.411412 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:22.411419 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:22.412231 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:22.412234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:22.412834 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:22.412842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:22.413289 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:22.413293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:22.413374 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:22.413958 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:22.413971 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:22.416073 (+ 2102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":93,"threads_started":1}
W20260619 18:38:22.416397 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:22.410398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46077 (local address 127.0.98.62:35899)
0619 18:38:22.410660 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:22.410663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:22.410738 (+ 75us) server_negotiation.cc:407] Connection header received
0619 18:38:22.410890 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:22.410894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:22.410950 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:22.411026 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:22.411525 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:22.412128 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:22.412943 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:22.413192 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:22.413276 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:22.416261 (+ 2985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46077: BlockingRecv error: recv got EOF from 127.0.98.1:46077 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
Jun 19 18:38:22 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:22 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:22.607764 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:22.602082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:39489)
0619 18:38:22.602400 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:22.602415 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:22.602526 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:22.602737 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:22.602740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:22.602966 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:22.603087 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:22.603094 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:22.603941 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:22.603945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:22.604627 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:22.604637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:22.605070 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:22.605072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:22.605136 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:22.605681 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:22.605698 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:22.607576 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":95,"threads_started":1}
W20260619 18:38:22.607863 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:22.602129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39489 (local address 127.0.98.62:35899)
0619 18:38:22.602322 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:22.602328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:22.602476 (+ 148us) server_negotiation.cc:407] Connection header received
0619 18:38:22.602593 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:22.602595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:22.602641 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:22.602703 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:22.603174 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:22.603816 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:22.604744 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:22.604979 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:22.605054 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:22.607752 (+ 2698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39489: BlockingRecv error: recv got EOF from 127.0.98.2:39489 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 19 18:38:22 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:22 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:22.804876 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:22.798501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:33369)
0619 18:38:22.798856 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:22.798870 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:22.798983 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:22.799219 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:22.799222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:22.799438 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:22.799562 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:22.799567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:22.800450 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:22.800453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:22.801074 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:22.801083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:22.801613 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:22.801616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:22.801689 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:22.802440 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:22.802454 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:22.804693 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
W20260619 18:38:22.804958 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:22.798679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33369 (local address 127.0.98.62:35899)
0619 18:38:22.798842 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:22.798846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:22.798963 (+ 117us) server_negotiation.cc:407] Connection header received
0619 18:38:22.799062 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:22.799065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:22.799120 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:22.799187 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:22.799695 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:22.800327 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:22.801245 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:22.801483 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:22.801575 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:22.804848 (+ 3273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33369: BlockingRecv error: recv got EOF from 127.0.98.3:33369 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 18:38:23 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:23 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:23.423830 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:23.417464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:40013)
0619 18:38:23.417816 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:23.417834 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:23.417988 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:23.418327 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:23.418329 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:23.418577 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:23.418698 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:23.418708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:23.419730 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:23.419734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:23.420553 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:23.420564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:23.421019 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:23.421022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:23.421092 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:23.421677 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:23.421693 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:23.423685 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
W20260619 18:38:23.424031 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:23.417602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40013 (local address 127.0.98.62:35899)
0619 18:38:23.417867 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:23.417871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:23.417921 (+ 50us) server_negotiation.cc:407] Connection header received
0619 18:38:23.418122 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:23.418126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:23.418182 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:23.418278 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:23.418860 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:23.419603 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:23.420668 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:23.420918 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:23.420999 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:23.423873 (+ 2874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40013: BlockingRecv error: recv got EOF from 127.0.98.1:40013 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
Jun 19 18:38:23 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:23 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:23.614948 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:23.608687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:43337)
0619 18:38:23.609026 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:23.609043 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:23.609181 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:23.609539 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:23.609542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:23.609804 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:23.609972 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:23.609981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:23.610944 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:23.610947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:23.611555 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:23.611563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:23.612006 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:23.612009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:23.612079 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:23.612658 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:23.612673 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:23.614767 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":123,"threads_started":1}
W20260619 18:38:23.615072 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:23.608798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43337 (local address 127.0.98.62:35899)
0619 18:38:23.608987 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:23.608992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:23.609122 (+ 130us) server_negotiation.cc:407] Connection header received
0619 18:38:23.609332 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:23.609336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:23.609399 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:23.609486 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:23.610103 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:23.610840 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:23.611662 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:23.611905 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:23.611992 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:23.614963 (+ 2971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43337: BlockingRecv error: recv got EOF from 127.0.98.2:43337 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 18:38:23 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:23 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:23.811491 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:23.805762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:46895)
0619 18:38:23.806110 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:23.806129 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:23.806246 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:23.806480 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:23.806482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:23.806690 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:23.806804 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:23.806809 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:23.807612 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:23.807614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:23.808205 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:23.808212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:23.808678 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:23.808680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:23.808748 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:23.809273 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:23.809284 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:23.811347 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":157,"threads_started":1}
W20260619 18:38:23.811643 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:23.805803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46895 (local address 127.0.98.62:35899)
0619 18:38:23.805972 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:23.805977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:23.806204 (+ 227us) server_negotiation.cc:407] Connection header received
0619 18:38:23.806323 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:23.806325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:23.806381 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:23.806454 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:23.806892 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:23.807527 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:23.808306 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:23.808573 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:23.808653 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:23.811512 (+ 2859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46895: BlockingRecv error: recv got EOF from 127.0.98.3:46895 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 18:38:24 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:24 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:24.430784 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:24.424739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:54075)
0619 18:38:24.425066 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:24.425085 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:24.425206 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:24.425464 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:24.425466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:24.425670 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:24.425783 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:24.425788 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:24.426651 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:24.426654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:24.427428 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:24.427442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:24.427891 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:24.427895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:24.427973 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:24.428528 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:24.428542 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:24.430641 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":100,"threads_started":1}
W20260619 18:38:24.430961 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:24.424904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54075 (local address 127.0.98.62:35899)
0619 18:38:24.425181 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:24.425185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:24.425200 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:38:24.425296 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:24.425298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:24.425345 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:24.425427 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:24.425872 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:24.426525 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:24.427546 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:24.427789 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:24.427887 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:24.430795 (+ 2908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54075: BlockingRecv error: recv got EOF from 127.0.98.1:54075 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":95,"threads_started":1}
Jun 19 18:38:24 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:24 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:24.621860 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:24.615923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:51081)
0619 18:38:24.616272 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:24.616289 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:24.616438 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:24.616797 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:24.616800 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:24.617084 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:24.617227 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:24.617236 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:24.618035 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:24.618038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:24.618621 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:24.618628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:24.619012 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:24.619014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:24.619074 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:24.619635 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:24.619647 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:24.621679 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":138,"threads_started":1}
W20260619 18:38:24.622123 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:24.615973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51081 (local address 127.0.98.62:35899)
0619 18:38:24.616139 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:24.616144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:24.616377 (+ 233us) server_negotiation.cc:407] Connection header received
0619 18:38:24.616577 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:24.616581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:24.616649 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:24.616729 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:24.617336 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:24.617949 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:24.618701 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:24.618917 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:24.619005 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:24.621845 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51081: BlockingRecv error: recv got EOF from 127.0.98.2:51081 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 18:38:24 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:24 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:24.818426 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:24.812355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:50687)
0619 18:38:24.812684 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:24.812697 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:24.812821 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:24.813079 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:24.813082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:24.813316 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:24.813426 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:24.813432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:24.814274 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:24.814278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:24.814861 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:24.814869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:24.815306 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:24.815310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:24.815396 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:24.816069 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:24.816086 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:24.818294 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":106,"threads_started":1}
W20260619 18:38:24.818542 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:24.812426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50687 (local address 127.0.98.62:35899)
0619 18:38:24.812575 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:24.812580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:24.812773 (+ 193us) server_negotiation.cc:407] Connection header received
0619 18:38:24.812914 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:24.812917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:24.812970 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:24.813044 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:24.813538 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:24.814173 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:24.814964 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:24.815198 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:24.815283 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:24.818430 (+ 3147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50687: BlockingRecv error: recv got EOF from 127.0.98.3:50687 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 18:38:25 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:25 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:25.437968 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:25.431751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:55337)
0619 18:38:25.432182 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:25.432198 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:25.432322 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:25.432602 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:25.432605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:25.432835 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:25.432942 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:25.432948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:25.433777 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:25.433781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:25.434382 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:25.434391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:25.434832 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:25.434836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:25.434916 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:25.435478 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:25.435498 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:25.437753 (+ 2255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":103,"threads_started":1}
W20260619 18:38:25.438141 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:25.431810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55337 (local address 127.0.98.62:35899)
0619 18:38:25.432095 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:25.432100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:25.432273 (+ 173us) server_negotiation.cc:407] Connection header received
0619 18:38:25.432416 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:25.432419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:25.432475 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:25.432567 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:25.433059 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:25.433659 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:25.434493 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:25.434733 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:25.434819 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:25.437970 (+ 3151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55337: BlockingRecv error: recv got EOF from 127.0.98.1:55337 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
Jun 19 18:38:25 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:25 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:25.628237 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:25.622764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:42097)
0619 18:38:25.623051 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:25.623066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:25.623190 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:25.623398 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:25.623400 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:25.623603 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:25.623714 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:25.623720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:25.624477 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:25.624481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:25.625109 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:25.625119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:25.625490 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:25.625492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:25.625552 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:25.626093 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:25.626106 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:25.628062 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":106,"threads_started":1}
W20260619 18:38:25.628346 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:25.622856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42097 (local address 127.0.98.62:35899)
0619 18:38:25.622999 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:25.623002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:25.623142 (+ 140us) server_negotiation.cc:407] Connection header received
0619 18:38:25.623257 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:25.623259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:25.623307 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:25.623374 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:25.623801 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:25.624370 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:25.625198 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:25.625413 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:25.625479 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:25.628237 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42097: BlockingRecv error: recv got EOF from 127.0.98.2:42097 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 18:38:25 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:25 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:25.824805 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:25.819254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:34475)
0619 18:38:25.819537 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:25.819550 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:25.819671 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:25.819911 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:25.819913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:25.820124 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:25.820238 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:25.820243 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:25.821030 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:25.821032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:25.821641 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:25.821649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:25.822099 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:25.822102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:25.822169 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:25.822685 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:25.822697 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:25.824667 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
W20260619 18:38:25.824945 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:25.819297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34475 (local address 127.0.98.62:35899)
0619 18:38:25.819431 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:25.819434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:25.819625 (+ 191us) server_negotiation.cc:407] Connection header received
0619 18:38:25.819762 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:25.819765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:25.819820 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:25.819879 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:25.820321 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:25.820933 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:25.821725 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:25.822004 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:25.822079 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:25.824823 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34475: BlockingRecv error: recv got EOF from 127.0.98.3:34475 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 18:38:26 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:26 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:26.445036 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:26.439131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:35483)
0619 18:38:26.439478 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:26.439493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:26.439609 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:26.439870 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:26.439873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:26.440075 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:26.440187 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:26.440193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:26.440977 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:26.440980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:26.441567 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:26.441575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:26.442051 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:26.442054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:26.442137 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:26.442739 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:26.442751 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:26.444862 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
W20260619 18:38:26.445181 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:26.439258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35483 (local address 127.0.98.62:35899)
0619 18:38:26.439593 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:26.439597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:26.439611 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:38:26.439685 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:26.439688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:26.439734 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:26.439847 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:26.440278 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:26.440889 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:26.441653 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:26.441937 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:26.442011 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:26.445016 (+ 3005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35483: BlockingRecv error: recv got EOF from 127.0.98.1:35483 (error 108)
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":79,"threads_started":1}
Jun 19 18:38:26 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:26 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:26.635587 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:26.629192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:48489)
0619 18:38:26.629493 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:26.629507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:26.629619 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:26.629983 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:26.629985 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:26.630227 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:26.630342 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:26.630348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:26.631234 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:26.631237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:26.631885 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:26.631893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:26.632397 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:26.632401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:26.632472 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:26.633036 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:26.633052 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:26.635385 (+ 2333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":110,"threads_started":1}
W20260619 18:38:26.635699 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:26.629274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48489 (local address 127.0.98.62:35899)
0619 18:38:26.629443 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:26.629446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:26.629573 (+ 127us) server_negotiation.cc:407] Connection header received
0619 18:38:26.629697 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:26.629699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:26.629749 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:26.629828 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:26.630482 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:26.631120 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:26.632015 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:26.632271 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:26.632361 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:26.635545 (+ 3184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48489: BlockingRecv error: recv got EOF from 127.0.98.2:48489 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 19 18:38:26 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:26 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:26.835345 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:26.825709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.3:58875)
0619 18:38:26.826029 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:26.826042 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:26.826159 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:26.826399 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:26.826402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:26.826613 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:26.826734 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:26.826740 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:26.827583 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:26.827587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:26.828217 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:26.828226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:26.828694 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:26.828697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:26.828780 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:26.832977 (+ 4197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:26.832991 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:26.835153 (+ 2162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":90,"threads_started":1}
W20260619 18:38:26.835539 9616 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:26.825821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58875 (local address 127.0.98.62:35899)
0619 18:38:26.826016 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:26.826020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:26.826110 (+ 90us) server_negotiation.cc:407] Connection header received
0619 18:38:26.826241 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:26.826244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:26.826296 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:26.826367 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:26.826854 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:26.827469 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:26.828330 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:26.828564 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:26.828650 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:26.835398 (+ 6748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58875: BlockingRecv error: recv got EOF from 127.0.98.3:58875 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 18:38:27 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:27 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:27.452580 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:27.446378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.1:57811)
0619 18:38:27.446742 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:27.446759 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:27.446882 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:27.447222 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:27.447225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:27.447524 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:27.447659 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:27.447670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:27.448617 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:27.448621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:27.449256 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:27.449265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:27.449721 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:27.449724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:27.449831 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:27.450446 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:27.450458 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:27.452421 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":105,"threads_started":1}
W20260619 18:38:27.452699 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:27.446561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57811 (local address 127.0.98.62:35899)
0619 18:38:27.446803 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:27.446808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:27.446825 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:38:27.446991 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:27.446994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:27.447052 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:27.447155 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:27.447795 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:27.448495 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:27.449370 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:27.449615 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:27.449693 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:27.452575 (+ 2882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57811: BlockingRecv error: recv got EOF from 127.0.98.1:57811 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":71,"threads_started":1}
Jun 19 18:38:27 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 18:38:27 dist-test-slave-rxv6 krb5kdc[8562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 18:38:27.642318 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:27.636600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35899 (local address 127.0.98.2:36599)
0619 18:38:27.636905 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:27.636919 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:27.637034 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:27.637318 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:27.637320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:27.637559 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:27.637684 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:27.637691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:27.638514 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:27.638517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:27.639105 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:27.639114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:27.639605 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:27.639608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:27.639675 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:27.640217 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:27.640228 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:27.642146 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35899: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":107,"threads_started":1}
W20260619 18:38:27.642419 9620 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:27.636674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36599 (local address 127.0.98.62:35899)
0619 18:38:27.636827 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:27.636831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:27.636997 (+ 166us) server_negotiation.cc:407] Connection header received
0619 18:38:27.637108 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:27.637111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:27.637162 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:27.637231 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:27.637791 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:27.638403 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:27.639236 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:27.639488 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:27.639581 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:27.642304 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36599: BlockingRecv error: recv got EOF from 127.0.98.2:36599 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:38:27.714397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 9145
I20260619 18:38:27.720567 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 9297
I20260619 18:38:27.725826 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 9429
I20260619 18:38:27.731017 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 9075
2026-06-19T18:38:27Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17889 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T18:38:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:38:27Z Disabled control of system clock
I20260619 18:38:27.760885 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33569
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 18:38:27.889928 9625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:27.890249 9625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:27.890285 9625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:27.894302 9625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:38:27.894366 9625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:27.894379 9625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:27.894394 9625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:38:27.894407 9625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:38:27.899268 9625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33569
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:27.900617 9625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:27.902377 9625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:27.912329 9633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:27.912323 9630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:27.912377 9625 server_base.cc:1034] running on GCE node
W20260619 18:38:27.912323 9631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:27.913400 9625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:27.914892 9625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:27.916064 9625 hybrid_clock.cc:648] HybridClock initialized: now 1781894307916040 us; error 42 us; skew 500 ppm
I20260619 18:38:27.918390 9625 webserver.cc:460] Webserver started at http://127.0.98.62:35535/ using document root <none> and password file <none>
I20260619 18:38:27.919140 9625 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:27.919216 9625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:27.919458 9625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:27.921954 9625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "010d5d5acd1446bcbf55b4ffb02476b3"
format_stamp: "Formatted at 2026-06-19 18:38:27 on dist-test-slave-rxv6"
server_key: "9c590012192a52ee18cd850798a628f1"
server_key_iv: "fe28c6a217f47c155bfe8cd88a2603ae"
server_key_version: "encryptionkey@0"
I20260619 18:38:27.922576 9625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "010d5d5acd1446bcbf55b4ffb02476b3"
format_stamp: "Formatted at 2026-06-19 18:38:27 on dist-test-slave-rxv6"
server_key: "9c590012192a52ee18cd850798a628f1"
server_key_iv: "fe28c6a217f47c155bfe8cd88a2603ae"
server_key_version: "encryptionkey@0"
I20260619 18:38:27.926944 9625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:38:27.930182 9639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:27.931704 9625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:38:27.931850 9625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "010d5d5acd1446bcbf55b4ffb02476b3"
format_stamp: "Formatted at 2026-06-19 18:38:27 on dist-test-slave-rxv6"
server_key: "9c590012192a52ee18cd850798a628f1"
server_key_iv: "fe28c6a217f47c155bfe8cd88a2603ae"
server_key_version: "encryptionkey@0"
I20260619 18:38:27.931974 9625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:27.945855 9625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:27.946720 9625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:27.946905 9625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:27.954639 9625 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33569
I20260619 18:38:27.954658 9691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33569 every 8 connection(s)
I20260619 18:38:27.955816 9625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:38:27.957428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 9625
I20260619 18:38:27.957525 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:38:27.957842 392 external_mini_cluster.cc:1428] Setting key b6732a38330078c432e7af2db28c02db
I20260619 18:38:27.959934 9692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:27.967526 9692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3: Bootstrap starting.
I20260619 18:38:27.968243 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:27.959427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.0.98.62:33569)
0619 18:38:27.960106 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:27.960124 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:27.960164 (+ 40us) server_negotiation.cc:407] Connection header received
0619 18:38:27.960916 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:27.960938 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:27.961426 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:27.961768 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:27.962275 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:27.963400 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:27.964646 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:27.965148 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:27.965282 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:27.966288 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:27.966294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:27.966310 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:38:27.966798 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:27.966865 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:38:27.966886 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:27.966966 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:27.967121 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 18:38:27.967377 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":126,"threads_started":1}
I20260619 18:38:27.970224 9692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:27.971133 9692 log.cc:824] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:27.973301 9692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3: No bootstrap required, opened a new log
I20260619 18:38:27.977394 9692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010d5d5acd1446bcbf55b4ffb02476b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33569 } }
I20260619 18:38:27.977664 9692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:27.977736 9692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 010d5d5acd1446bcbf55b4ffb02476b3, State: Initialized, Role: FOLLOWER
I20260619 18:38:27.978258 9692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010d5d5acd1446bcbf55b4ffb02476b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33569 } }
I20260619 18:38:27.978405 9692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:38:27.978474 9692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:38:27.978580 9692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:27.979590 9692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010d5d5acd1446bcbf55b4ffb02476b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33569 } }
I20260619 18:38:27.979972 9692 leader_election.cc:302] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 010d5d5acd1446bcbf55b4ffb02476b3; no voters:
I20260619 18:38:27.980355 9692 leader_election.cc:288] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:38:27.980461 9697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:38:27.980671 9697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [term 1 LEADER]: Becoming Leader. State: Replica: 010d5d5acd1446bcbf55b4ffb02476b3, State: Running, Role: LEADER
I20260619 18:38:27.981070 9697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010d5d5acd1446bcbf55b4ffb02476b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33569 } }
I20260619 18:38:27.981832 9692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:38:27.982946 9699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 010d5d5acd1446bcbf55b4ffb02476b3. Latest consensus state: current_term: 1 leader_uuid: "010d5d5acd1446bcbf55b4ffb02476b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010d5d5acd1446bcbf55b4ffb02476b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33569 } } }
I20260619 18:38:27.983080 9699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:27.983368 9698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "010d5d5acd1446bcbf55b4ffb02476b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "010d5d5acd1446bcbf55b4ffb02476b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33569 } } }
I20260619 18:38:27.983469 9698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:27.983747 9706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:38:27.987210 9706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:38:27.993661 9706 catalog_manager.cc:1347] Generated new cluster ID: ab2f33e2299342c69f18710fbe9d728b
I20260619 18:38:27.993734 9706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:38:28.003119 9706 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:38:28.004551 9706 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:38:28.014923 9706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 010d5d5acd1446bcbf55b4ffb02476b3: Generated new TSK 0
I20260619 18:38:28.016090 9706 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:38:28.022044 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33569
--builtin_ntp_servers=127.0.98.20:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 18:38:28.152011 9716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:28.152362 9716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:28.152415 9716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:28.156939 9716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:28.157034 9716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:28.157119 9716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:38:28.162328 9716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:28.163805 9716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:28.165576 9716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:28.176947 9724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:28.176939 9722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:28.176939 9721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:28.177242 9716 server_base.cc:1034] running on GCE node
I20260619 18:38:28.177809 9716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:28.178645 9716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:28.179822 9716 hybrid_clock.cc:648] HybridClock initialized: now 1781894308179797 us; error 43 us; skew 500 ppm
I20260619 18:38:28.182096 9716 webserver.cc:460] Webserver started at http://127.0.98.1:44829/ using document root <none> and password file <none>
I20260619 18:38:28.182852 9716 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:28.182929 9716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:28.183248 9716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:28.185792 9716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "ed1c6564ad5744f29de131c80145f0d7"
format_stamp: "Formatted at 2026-06-19 18:38:28 on dist-test-slave-rxv6"
server_key: "c2fb9d8f332e9d5c904d12f383b5ee2b"
server_key_iv: "07ff186698977be641aebed1e805ef88"
server_key_version: "encryptionkey@0"
I20260619 18:38:28.186476 9716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ed1c6564ad5744f29de131c80145f0d7"
format_stamp: "Formatted at 2026-06-19 18:38:28 on dist-test-slave-rxv6"
server_key: "c2fb9d8f332e9d5c904d12f383b5ee2b"
server_key_iv: "07ff186698977be641aebed1e805ef88"
server_key_version: "encryptionkey@0"
I20260619 18:38:28.191099 9716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:38:28.194229 9730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:28.195698 9716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:38:28.195823 9716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "ed1c6564ad5744f29de131c80145f0d7"
format_stamp: "Formatted at 2026-06-19 18:38:28 on dist-test-slave-rxv6"
server_key: "c2fb9d8f332e9d5c904d12f383b5ee2b"
server_key_iv: "07ff186698977be641aebed1e805ef88"
server_key_version: "encryptionkey@0"
I20260619 18:38:28.195952 9716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:28.211010 9716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:28.211946 9716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:28.212162 9716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:28.212901 9716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:28.214116 9716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:28.214169 9716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:28.214228 9716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:28.214252 9716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:28.224376 9716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43433
I20260619 18:38:28.224399 9843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43433 every 8 connection(s)
I20260619 18:38:28.225535 9716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:38:28.230217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 9716
I20260619 18:38:28.230360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:38:28.230724 392 external_mini_cluster.cc:1428] Setting key e8d1b7a51904b776ba6738d9a99fc401
I20260619 18:38:28.233150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33569
--builtin_ntp_servers=127.0.98.20:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 18:38:28.236897 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:28.227819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54051 (local address 127.0.98.62:33569)
0619 18:38:28.227961 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:28.227965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:28.229370 (+ 1405us) server_negotiation.cc:407] Connection header received
0619 18:38:28.231166 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:28.231170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:28.231244 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:28.231328 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:28.232748 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:28.233344 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:28.234671 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:28.234905 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:28.234998 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:28.236334 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:28.236337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:28.236339 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:38:28.236355 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:28.236377 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:38:28.236385 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:28.236447 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:28.236600 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 18:38:28.236700 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:38:28.238025 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:28.228377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33569 (local address 127.0.98.1:54051)
0619 18:38:28.229190 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:28.229240 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:28.230890 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:28.231577 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:28.231588 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:28.231760 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:28.232596 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:28.232617 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:28.233491 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:28.233494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:28.234544 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:28.234554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:28.235073 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:28.235077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:28.235262 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:28.236045 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:38:28.236046 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:28.236098 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:38:28.236178 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:28.236454 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:28.236464 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:28.236471 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 18:38:28.236598 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 18:38:28.236983 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":134,"threads_started":1}
I20260619 18:38:28.239531 9844 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33569
I20260619 18:38:28.239792 9844 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:28.240411 9844 heartbeater.cc:511] Master 127.0.98.62:33569 requested a full tablet report, sending...
I20260619 18:38:28.242277 9656 ts_manager.cc:194] Registered new tserver with Master: ed1c6564ad5744f29de131c80145f0d7 (127.0.98.1:43433)
I20260619 18:38:28.243615 9656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:54051
W20260619 18:38:28.362486 9847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:28.362848 9847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:28.362905 9847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:28.367197 9847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:28.367276 9847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:28.367358 9847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:38:28.372591 9847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:28.374148 9847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:28.375895 9847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:28.387470 9852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:28.387455 9853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:28.387446 9855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:28.387670 9847 server_base.cc:1034] running on GCE node
I20260619 18:38:28.388139 9847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:28.388934 9847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:28.390100 9847 hybrid_clock.cc:648] HybridClock initialized: now 1781894308390067 us; error 46 us; skew 500 ppm
I20260619 18:38:28.392145 9847 webserver.cc:460] Webserver started at http://127.0.98.2:36643/ using document root <none> and password file <none>
I20260619 18:38:28.392817 9847 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:28.392886 9847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:28.393126 9847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:28.395576 9847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "ebcd0f50a42047a09fd32db6e00d1a37"
format_stamp: "Formatted at 2026-06-19 18:38:28 on dist-test-slave-rxv6"
server_key: "37299137c862e88f6e7a8ba4bdb8bf48"
server_key_iv: "124eb41c6ef0781346fae36435893104"
server_key_version: "encryptionkey@0"
I20260619 18:38:28.396189 9847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ebcd0f50a42047a09fd32db6e00d1a37"
format_stamp: "Formatted at 2026-06-19 18:38:28 on dist-test-slave-rxv6"
server_key: "37299137c862e88f6e7a8ba4bdb8bf48"
server_key_iv: "124eb41c6ef0781346fae36435893104"
server_key_version: "encryptionkey@0"
I20260619 18:38:28.400506 9847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:38:28.403683 9861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:28.405370 9847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:38:28.405503 9847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "ebcd0f50a42047a09fd32db6e00d1a37"
format_stamp: "Formatted at 2026-06-19 18:38:28 on dist-test-slave-rxv6"
server_key: "37299137c862e88f6e7a8ba4bdb8bf48"
server_key_iv: "124eb41c6ef0781346fae36435893104"
server_key_version: "encryptionkey@0"
I20260619 18:38:28.405627 9847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:28.447052 9847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:28.447978 9847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:28.448176 9847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:28.448877 9847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:28.450034 9847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:28.450084 9847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:28.450141 9847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:28.450166 9847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:28.459993 9847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40543
I20260619 18:38:28.460026 9974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40543 every 8 connection(s)
I20260619 18:38:28.461177 9847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:38:28.471421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 9847
I20260619 18:38:28.471541 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:38:28.471782 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:28.463449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57585 (local address 127.0.98.62:33569)
0619 18:38:28.463619 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:28.463623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:28.465017 (+ 1394us) server_negotiation.cc:407] Connection header received
0619 18:38:28.466635 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:28.466639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:28.466708 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:28.466791 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:28.467929 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:28.468502 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:28.469747 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:28.470003 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:28.470099 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:28.471271 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:28.471275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:28.471277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:38:28.471290 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:28.471313 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:38:28.471322 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:28.471383 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:28.471543 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 18:38:28.471617 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:38:28.471844 392 external_mini_cluster.cc:1428] Setting key 1d03bb1de248c2a54450a18e97929562
I20260619 18:38:28.472944 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:28.463967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33569 (local address 127.0.98.2:57585)
0619 18:38:28.464813 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:28.464860 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:28.466378 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:28.467011 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:28.467021 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:28.467132 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:28.467763 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:28.467778 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:28.468629 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:28.468632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:28.469606 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:28.469615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:28.470157 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:28.470161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:28.470338 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:28.470992 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:38:28.470994 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:28.471043 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:38:28.471115 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:28.471392 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:28.471401 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:28.471408 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 18:38:28.471535 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 18:38:28.471865 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":142,"threads_started":1}
I20260619 18:38:28.474047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33569
--builtin_ntp_servers=127.0.98.20:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 18:38:28.474419 9975 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33569
I20260619 18:38:28.474700 9975 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:28.475363 9975 heartbeater.cc:511] Master 127.0.98.62:33569 requested a full tablet report, sending...
I20260619 18:38:28.476594 9656 ts_manager.cc:194] Registered new tserver with Master: ebcd0f50a42047a09fd32db6e00d1a37 (127.0.98.2:40543)
I20260619 18:38:28.477370 9656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:57585
W20260619 18:38:28.602824 9978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:28.603219 9978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:28.603264 9978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:28.608521 9978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:28.608629 9978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:28.608700 9978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:38:28.614333 9978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:28.615803 9978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:28.617646 9978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:28.629738 9984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:28.629865 9978 server_base.cc:1034] running on GCE node
W20260619 18:38:28.629766 9983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:28.630115 9986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:28.630599 9978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:28.631448 9978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:28.632627 9978 hybrid_clock.cc:648] HybridClock initialized: now 1781894308632591 us; error 49 us; skew 500 ppm
I20260619 18:38:28.634943 9978 webserver.cc:460] Webserver started at http://127.0.98.3:40205/ using document root <none> and password file <none>
I20260619 18:38:28.635689 9978 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:28.635767 9978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:28.636031 9978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:28.638682 9978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "9d273acaa1cb4a9fa92fcb9e1212a95b"
format_stamp: "Formatted at 2026-06-19 18:38:28 on dist-test-slave-rxv6"
server_key: "708d02017a040faafd8add9485fa4cae"
server_key_iv: "26e030002ea11ef15e62161415d8fd6d"
server_key_version: "encryptionkey@0"
I20260619 18:38:28.639303 9978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9d273acaa1cb4a9fa92fcb9e1212a95b"
format_stamp: "Formatted at 2026-06-19 18:38:28 on dist-test-slave-rxv6"
server_key: "708d02017a040faafd8add9485fa4cae"
server_key_iv: "26e030002ea11ef15e62161415d8fd6d"
server_key_version: "encryptionkey@0"
I20260619 18:38:28.643915 9978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:38:28.647118 9992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:28.648813 9978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:38:28.648939 9978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "9d273acaa1cb4a9fa92fcb9e1212a95b"
format_stamp: "Formatted at 2026-06-19 18:38:28 on dist-test-slave-rxv6"
server_key: "708d02017a040faafd8add9485fa4cae"
server_key_iv: "26e030002ea11ef15e62161415d8fd6d"
server_key_version: "encryptionkey@0"
I20260619 18:38:28.649096 9978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:28.678503 9978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:28.679574 9978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:28.679805 9978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:28.680639 9978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:28.682013 9978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:28.682075 9978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:28.682126 9978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:28.682154 9978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:28.692911 9978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39439
I20260619 18:38:28.692974 10105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39439 every 8 connection(s)
I20260619 18:38:28.694161 9978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:38:28.702615 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 9978
I20260619 18:38:28.702732 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:38:28.703042 392 external_mini_cluster.cc:1428] Setting key 5aa7282b502e2580d7a0f7beafd06684
I20260619 18:38:28.705801 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:28.696557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43695 (local address 127.0.98.62:33569)
0619 18:38:28.696729 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:28.696733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:28.698392 (+ 1659us) server_negotiation.cc:407] Connection header received
0619 18:38:28.700052 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:28.700056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:28.700121 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:28.700210 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:28.701411 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:28.702097 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:28.703490 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:28.703774 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:28.703878 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:28.705231 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:28.705235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:28.705239 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:38:28.705255 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:28.705283 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:38:28.705293 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:28.705369 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:28.705553 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 18:38:28.705632 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:38:28.706944 10108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:28.697207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33569 (local address 127.0.98.3:43695)
0619 18:38:28.698196 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:28.698249 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:28.699793 (+ 1544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:28.700412 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:28.700421 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:28.700533 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:28.701247 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:28.701265 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:28.702223 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:28.702228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:28.703354 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:28.703367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:28.703948 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:28.703952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:28.704180 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:28.704913 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:38:28.704915 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:28.704976 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:38:28.705069 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:28.705382 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:28.705390 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:28.705396 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:38:28.705536 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 18:38:28.705945 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":158,"threads_started":1}
I20260619 18:38:28.708210 10106 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33569
I20260619 18:38:28.708487 10106 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:28.709136 10106 heartbeater.cc:511] Master 127.0.98.62:33569 requested a full tablet report, sending...
I20260619 18:38:28.710361 9656 ts_manager.cc:194] Registered new tserver with Master: 9d273acaa1cb4a9fa92fcb9e1212a95b (127.0.98.3:39439)
I20260619 18:38:28.711086 9656 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:43695
I20260619 18:38:28.717592 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 18:38:28.721058 9695 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:28.719932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51412 (local address 127.0.98.62:33569)
0619 18:38:28.720102 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:28.720106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:28.720327 (+ 221us) server_negotiation.cc:407] Connection header received
0619 18:38:28.720436 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:28.720438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:28.720489 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:28.720602 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:28.720903 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51412: BlockingRecv error: recv got EOF from 127.0.0.1:51412 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:38:28.721160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 9716
I20260619 18:38:28.727334 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 9847
I20260619 18:38:28.733258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 9978
I20260619 18:38:28.739292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 9625
2026-06-19T18:38:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1008 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T18:38:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:38:28Z Disabled control of system clock
I20260619 18:38:28.770954 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38007
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 18:38:28.896862 10117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:28.897203 10117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:28.897241 10117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:28.901284 10117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:38:28.901362 10117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:28.901376 10117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:28.901393 10117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:38:28.901407 10117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:38:28.906895 10117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38007
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:28.908406 10117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:28.910310 10117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:28.920434 10123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:28.920425 10122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:28.920509 10117 server_base.cc:1034] running on GCE node
W20260619 18:38:28.920434 10125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:28.921535 10117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:28.922993 10117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:28.924172 10117 hybrid_clock.cc:648] HybridClock initialized: now 1781894308924149 us; error 37 us; skew 500 ppm
I20260619 18:38:28.926426 10117 webserver.cc:460] Webserver started at http://127.0.98.62:41301/ using document root <none> and password file <none>
I20260619 18:38:28.927207 10117 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:28.927290 10117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:28.927543 10117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:28.930101 10117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "09db1ff89eeb42ffa1a5c999df9d3ad8"
format_stamp: "Formatted at 2026-06-19 18:38:28 on dist-test-slave-rxv6"
server_key: "845352716addb6b58ef2a98d72a93572"
server_key_iv: "0f89acab4f2a048922b898c93891c608"
server_key_version: "encryptionkey@0"
I20260619 18:38:28.930732 10117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "09db1ff89eeb42ffa1a5c999df9d3ad8"
format_stamp: "Formatted at 2026-06-19 18:38:28 on dist-test-slave-rxv6"
server_key: "845352716addb6b58ef2a98d72a93572"
server_key_iv: "0f89acab4f2a048922b898c93891c608"
server_key_version: "encryptionkey@0"
I20260619 18:38:28.935110 10117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:38:28.938086 10131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:28.939702 10117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:38:28.939833 10117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "09db1ff89eeb42ffa1a5c999df9d3ad8"
format_stamp: "Formatted at 2026-06-19 18:38:28 on dist-test-slave-rxv6"
server_key: "845352716addb6b58ef2a98d72a93572"
server_key_iv: "0f89acab4f2a048922b898c93891c608"
server_key_version: "encryptionkey@0"
I20260619 18:38:28.939947 10117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:28.945250 10117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:28.946039 10117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:28.946223 10117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:28.953899 10117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38007
I20260619 18:38:28.953928 10183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38007 every 8 connection(s)
I20260619 18:38:28.955078 10117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:38:28.957667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 10117
I20260619 18:38:28.957779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:38:28.958145 392 external_mini_cluster.cc:1428] Setting key ae79785b40f79c9fa4d883a758831f58
I20260619 18:38:28.959205 10184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:28.964844 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:28.959858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.0.98.62:38007)
0619 18:38:28.960678 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:28.960699 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:28.960745 (+ 46us) server_negotiation.cc:407] Connection header received
0619 18:38:28.961548 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:28.961571 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:28.962067 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:28.962443 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:28.963319 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:28.963325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:28.963340 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:38:28.963350 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:28.963416 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:38:28.963435 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:28.963598 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:28.963674 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 18:38:28.963966 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":143,"threads_started":1}
I20260619 18:38:28.966780 10184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8: Bootstrap starting.
I20260619 18:38:28.969565 10184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:28.970417 10184 log.cc:824] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:28.972769 10184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8: No bootstrap required, opened a new log
I20260619 18:38:28.976999 10184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09db1ff89eeb42ffa1a5c999df9d3ad8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } }
I20260619 18:38:28.977315 10184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:28.977372 10184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09db1ff89eeb42ffa1a5c999df9d3ad8, State: Initialized, Role: FOLLOWER
I20260619 18:38:28.977900 10184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09db1ff89eeb42ffa1a5c999df9d3ad8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } }
I20260619 18:38:28.978116 10184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:38:28.978189 10184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:38:28.978297 10184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:28.979313 10184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09db1ff89eeb42ffa1a5c999df9d3ad8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } }
I20260619 18:38:28.979667 10184 leader_election.cc:302] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09db1ff89eeb42ffa1a5c999df9d3ad8; no voters:
I20260619 18:38:28.979969 10184 leader_election.cc:288] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:38:28.980108 10189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:38:28.980346 10189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [term 1 LEADER]: Becoming Leader. State: Replica: 09db1ff89eeb42ffa1a5c999df9d3ad8, State: Running, Role: LEADER
I20260619 18:38:28.980754 10189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09db1ff89eeb42ffa1a5c999df9d3ad8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } }
I20260619 18:38:28.981420 10184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:38:28.982630 10191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09db1ff89eeb42ffa1a5c999df9d3ad8. Latest consensus state: current_term: 1 leader_uuid: "09db1ff89eeb42ffa1a5c999df9d3ad8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09db1ff89eeb42ffa1a5c999df9d3ad8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } }
I20260619 18:38:28.982618 10190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09db1ff89eeb42ffa1a5c999df9d3ad8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09db1ff89eeb42ffa1a5c999df9d3ad8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38007 } } }
I20260619 18:38:28.982764 10191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:28.982764 10190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:28.983207 10198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:38:28.986600 10198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:38:28.993472 10198 catalog_manager.cc:1347] Generated new cluster ID: 6cdfc9b07271404da83e24a00d9455ae
I20260619 18:38:28.993567 10198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:38:29.011832 10198 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:38:29.012247 10198 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:38:29.031061 10198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09db1ff89eeb42ffa1a5c999df9d3ad8: Generated new TSK 0
I20260619 18:38:29.032001 10198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:38:29.045760 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38007
--builtin_ntp_servers=127.0.98.20:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 18:38:29.172860 10208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:29.173192 10208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:29.173234 10208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:29.177353 10208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:29.177434 10208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:29.177500 10208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:38:29.182912 10208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:29.184334 10208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:29.186085 10208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:29.198169 10214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:29.198489 10213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:29.198629 10208 server_base.cc:1034] running on GCE node
W20260619 18:38:29.198838 10216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:29.199435 10208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:29.200343 10208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:29.201535 10208 hybrid_clock.cc:648] HybridClock initialized: now 1781894309201490 us; error 62 us; skew 500 ppm
I20260619 18:38:29.203886 10208 webserver.cc:460] Webserver started at http://127.0.98.1:34017/ using document root <none> and password file <none>
I20260619 18:38:29.204574 10208 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:29.204668 10208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:29.204918 10208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:29.207851 10208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "8bd0c9f307b246cea304ceb96eb270d7"
format_stamp: "Formatted at 2026-06-19 18:38:29 on dist-test-slave-rxv6"
server_key: "ee7ec0eb02f146508632e49ecf557b31"
server_key_iv: "5c6375fb6e5f8e2d917b7a61f427035c"
server_key_version: "encryptionkey@0"
I20260619 18:38:29.208668 10208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8bd0c9f307b246cea304ceb96eb270d7"
format_stamp: "Formatted at 2026-06-19 18:38:29 on dist-test-slave-rxv6"
server_key: "ee7ec0eb02f146508632e49ecf557b31"
server_key_iv: "5c6375fb6e5f8e2d917b7a61f427035c"
server_key_version: "encryptionkey@0"
I20260619 18:38:29.213418 10208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 18:38:29.216228 10222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:29.217690 10208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 18:38:29.217810 10208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "8bd0c9f307b246cea304ceb96eb270d7"
format_stamp: "Formatted at 2026-06-19 18:38:29 on dist-test-slave-rxv6"
server_key: "ee7ec0eb02f146508632e49ecf557b31"
server_key_iv: "5c6375fb6e5f8e2d917b7a61f427035c"
server_key_version: "encryptionkey@0"
I20260619 18:38:29.217960 10208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:29.223152 10208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:29.223829 10208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:29.223989 10208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:29.224573 10208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:29.225586 10208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:29.225633 10208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:29.225675 10208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:29.225703 10208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:29.235138 10208 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39843
I20260619 18:38:29.235162 10335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39843 every 8 connection(s)
I20260619 18:38:29.236164 10208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:38:29.242908 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:29.238220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33157 (local address 127.0.98.62:38007)
0619 18:38:29.238375 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:29.238379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:29.239666 (+ 1287us) server_negotiation.cc:407] Connection header received
0619 18:38:29.241040 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:29.241044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:29.241094 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:29.241178 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:29.242440 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:29.242444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:29.242447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:38:29.242460 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:29.242486 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:38:29.242495 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:29.242576 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:29.242700 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 18:38:29.242750 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:38:29.243350 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 10208
I20260619 18:38:29.243458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:38:29.243798 392 external_mini_cluster.cc:1428] Setting key c454eac128db6c7aac18ceb4e57f511b
I20260619 18:38:29.243964 10338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:29.238721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38007 (local address 127.0.98.1:33157)
0619 18:38:29.239499 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:29.239544 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:29.240821 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:29.241366 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:29.241374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:29.241477 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:29.242165 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:38:29.242166 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:29.242213 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:38:29.242288 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:29.242564 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:29.242571 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:29.242577 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:38:29.242747 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 18:38:29.243011 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":147,"threads_started":1}
I20260619 18:38:29.245236 10336 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38007
I20260619 18:38:29.245494 10336 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:29.246021 10336 heartbeater.cc:511] Master 127.0.98.62:38007 requested a full tablet report, sending...
I20260619 18:38:29.246021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38007
--builtin_ntp_servers=127.0.98.20:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 18:38:29.247972 10148 ts_manager.cc:194] Registered new tserver with Master: 8bd0c9f307b246cea304ceb96eb270d7 (127.0.98.1:39843)
W20260619 18:38:29.372149 10339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:29.372494 10339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:29.372550 10339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:29.376708 10339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:29.376782 10339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:29.376868 10339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:38:29.381875 10339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:29.383385 10339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:29.385143 10339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:29.396452 10347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:29.396651 10339 server_base.cc:1034] running on GCE node
W20260619 18:38:29.396481 10345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:29.396466 10344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:29.397261 10339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:29.398051 10339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:29.399237 10339 hybrid_clock.cc:648] HybridClock initialized: now 1781894309399213 us; error 43 us; skew 500 ppm
I20260619 18:38:29.401455 10339 webserver.cc:460] Webserver started at http://127.0.98.2:36845/ using document root <none> and password file <none>
I20260619 18:38:29.402230 10339 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:29.402309 10339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:29.402563 10339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:29.405208 10339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "83b9b92481f34ddf97f830eec24fd8b0"
format_stamp: "Formatted at 2026-06-19 18:38:29 on dist-test-slave-rxv6"
server_key: "5245878422551e9c720edc7092d5a5de"
server_key_iv: "08e27cbad7be73e1eddeed2f6e900cfc"
server_key_version: "encryptionkey@0"
I20260619 18:38:29.405898 10339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "83b9b92481f34ddf97f830eec24fd8b0"
format_stamp: "Formatted at 2026-06-19 18:38:29 on dist-test-slave-rxv6"
server_key: "5245878422551e9c720edc7092d5a5de"
server_key_iv: "08e27cbad7be73e1eddeed2f6e900cfc"
server_key_version: "encryptionkey@0"
I20260619 18:38:29.411007 10339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 18:38:29.414150 10353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:29.415831 10339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:38:29.415977 10339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "83b9b92481f34ddf97f830eec24fd8b0"
format_stamp: "Formatted at 2026-06-19 18:38:29 on dist-test-slave-rxv6"
server_key: "5245878422551e9c720edc7092d5a5de"
server_key_iv: "08e27cbad7be73e1eddeed2f6e900cfc"
server_key_version: "encryptionkey@0"
I20260619 18:38:29.416138 10339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:29.422192 10339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:29.423054 10339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:29.423273 10339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:29.423981 10339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:29.425139 10339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:29.425194 10339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:29.425243 10339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:29.425276 10339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:29.436204 10339 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33131
I20260619 18:38:29.436280 10466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33131 every 8 connection(s)
I20260619 18:38:29.437439 10339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:38:29.443723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 10339
I20260619 18:38:29.443863 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:38:29.444320 392 external_mini_cluster.cc:1428] Setting key 786fadae087f34b65824f65ab8ff8ff4
I20260619 18:38:29.445510 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:29.439944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50369 (local address 127.0.98.62:38007)
0619 18:38:29.440109 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:29.440113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:29.441580 (+ 1467us) server_negotiation.cc:407] Connection header received
0619 18:38:29.443255 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:29.443260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:29.443321 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:29.443418 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:29.444966 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:29.444971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:29.444974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:38:29.444997 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:29.445026 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:38:29.445036 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:29.445155 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:29.445301 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 18:38:29.445354 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:38:29.446803 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:29.440527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38007 (local address 127.0.98.2:50369)
0619 18:38:29.441392 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:29.441445 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:29.442944 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:29.443721 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:29.443733 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:29.443863 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:29.444635 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:38:29.444637 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:29.444692 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:38:29.444786 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:29.445151 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:29.445161 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:29.445168 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 18:38:29.445359 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 18:38:29.445698 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":156,"threads_started":1}
I20260619 18:38:29.446846 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38007
--builtin_ntp_servers=127.0.98.20:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 18:38:29.448207 10467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38007
I20260619 18:38:29.448568 10467 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:29.449172 10467 heartbeater.cc:511] Master 127.0.98.62:38007 requested a full tablet report, sending...
I20260619 18:38:29.450603 10148 ts_manager.cc:194] Registered new tserver with Master: 83b9b92481f34ddf97f830eec24fd8b0 (127.0.98.2:33131)
W20260619 18:38:29.585742 10470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:29.586172 10470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:29.586225 10470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:29.591739 10470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:29.591845 10470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:29.591941 10470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:38:29.598623 10470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:29.600087 10470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:29.601893 10470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:29.614660 10476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:29.614660 10478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:29.614760 10475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:29.615723 10470 server_base.cc:1034] running on GCE node
I20260619 18:38:29.616302 10470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:29.617173 10470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:29.618345 10470 hybrid_clock.cc:648] HybridClock initialized: now 1781894309618320 us; error 45 us; skew 500 ppm
I20260619 18:38:29.620889 10470 webserver.cc:460] Webserver started at http://127.0.98.3:35961/ using document root <none> and password file <none>
I20260619 18:38:29.621675 10470 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:29.621755 10470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:29.622044 10470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:29.624800 10470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "4ce7329c5a9a4ed4a20d6e43015179f1"
format_stamp: "Formatted at 2026-06-19 18:38:29 on dist-test-slave-rxv6"
server_key: "42c9e5355ea000125a66141e3239f14b"
server_key_iv: "c5f345c487516e70eb7f02cd05690706"
server_key_version: "encryptionkey@0"
I20260619 18:38:29.625468 10470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4ce7329c5a9a4ed4a20d6e43015179f1"
format_stamp: "Formatted at 2026-06-19 18:38:29 on dist-test-slave-rxv6"
server_key: "42c9e5355ea000125a66141e3239f14b"
server_key_iv: "c5f345c487516e70eb7f02cd05690706"
server_key_version: "encryptionkey@0"
I20260619 18:38:29.630204 10470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 18:38:29.633335 10484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:29.635020 10470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:38:29.635154 10470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "4ce7329c5a9a4ed4a20d6e43015179f1"
format_stamp: "Formatted at 2026-06-19 18:38:29 on dist-test-slave-rxv6"
server_key: "42c9e5355ea000125a66141e3239f14b"
server_key_iv: "c5f345c487516e70eb7f02cd05690706"
server_key_version: "encryptionkey@0"
I20260619 18:38:29.635284 10470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:29.640744 10470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:29.641465 10470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:29.641638 10470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:29.642309 10470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:29.643355 10470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:29.643411 10470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:29.643465 10470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:29.643496 10470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:29.652952 10470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42443
I20260619 18:38:29.653018 10597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42443 every 8 connection(s)
I20260619 18:38:29.654137 10470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:38:29.655260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 10470
I20260619 18:38:29.655364 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:38:29.655668 392 external_mini_cluster.cc:1428] Setting key 68e3cf1f748a2a38704c3e341813db61
I20260619 18:38:29.661731 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:29.656369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45949 (local address 127.0.98.62:38007)
0619 18:38:29.656535 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:29.656540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:29.658211 (+ 1671us) server_negotiation.cc:407] Connection header received
0619 18:38:29.659685 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:29.659688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:29.659742 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:29.659816 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:29.661165 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:29.661169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:29.661171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:38:29.661185 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:29.661239 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:38:29.661249 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:29.661338 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:29.661527 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 18:38:29.661577 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:38:29.662882 10600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:29.657073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38007 (local address 127.0.98.3:45949)
0619 18:38:29.658055 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:29.658108 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:29.659434 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:29.660034 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:29.660042 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:29.660161 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:29.660850 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:38:29.660851 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:29.660902 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:38:29.660987 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:29.661356 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:29.661366 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:29.661372 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:38:29.661556 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 18:38:29.661826 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":144,"threads_started":1}
I20260619 18:38:29.664327 10598 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38007
I20260619 18:38:29.664608 10598 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:29.665162 10598 heartbeater.cc:511] Master 127.0.98.62:38007 requested a full tablet report, sending...
I20260619 18:38:29.666461 10148 ts_manager.cc:194] Registered new tserver with Master: 4ce7329c5a9a4ed4a20d6e43015179f1 (127.0.98.3:42443)
I20260619 18:38:29.669854 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 18:38:29.673002 10187 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:38:29.672072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47460 (local address 127.0.98.62:38007)
0619 18:38:29.672220 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:29.672224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:29.672429 (+ 205us) server_negotiation.cc:407] Connection header received
0619 18:38:29.672527 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:29.672530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:29.672569 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:29.672637 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:29.672859 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47460: BlockingRecv error: recv got EOF from 127.0.0.1:47460 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:38:29.673117 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 10208
I20260619 18:38:29.679327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 10339
I20260619 18:38:29.684902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 10470
I20260619 18:38:29.690068 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 10117
2026-06-19T18:38:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (949 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:38:29 dist-test-slave-rxv6 krb5kdc[10608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:38:29 dist-test-slave-rxv6 krb5kdc[10608](info): set up 2 sockets
Jun 19 18:38:29 dist-test-slave-rxv6 krb5kdc[10608](info): commencing operation
krb5kdc: starting...
W20260619 18:38:30.663105 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:38:30 dist-test-slave-rxv6 krb5kdc[10608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:38:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:38:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:38:30.785329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38865
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:30.908613 10621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:30.908969 10621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:30.909029 10621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:30.913272 10621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:38:30.913367 10621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:30.913396 10621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:30.913416 10621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:38:30.913437 10621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:38:30.919741 10621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:30.921214 10621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:30.922518 10621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:30.932736 10626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:30.932749 10629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:30.932883 10621 server_base.cc:1034] running on GCE node
W20260619 18:38:30.932752 10627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:30.933892 10621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:30.935456 10621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:30.936667 10621 hybrid_clock.cc:648] HybridClock initialized: now 1781894310936656 us; error 42 us; skew 500 ppm
Jun 19 18:38:30 dist-test-slave-rxv6 krb5kdc[10608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:30.940176 10621 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:38:30.941345 10621 webserver.cc:460] Webserver started at http://127.0.98.62:33197/ using document root <none> and password file <none>
I20260619 18:38:30.942056 10621 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:30.942127 10621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:30.942353 10621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:30.944916 10621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "d1acc47331ad42618f40946d11352890"
format_stamp: "Formatted at 2026-06-19 18:38:30 on dist-test-slave-rxv6"
server_key: "f77f47709bb5d6da16bae250d86232be"
server_key_iv: "df5d47ce8532b7c90fd2a76ea6986b96"
server_key_version: "encryptionkey@0"
I20260619 18:38:30.945567 10621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "d1acc47331ad42618f40946d11352890"
format_stamp: "Formatted at 2026-06-19 18:38:30 on dist-test-slave-rxv6"
server_key: "f77f47709bb5d6da16bae250d86232be"
server_key_iv: "df5d47ce8532b7c90fd2a76ea6986b96"
server_key_version: "encryptionkey@0"
I20260619 18:38:30.950222 10621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260619 18:38:30.953099 10636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:30.954727 10621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:38:30.954873 10621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "d1acc47331ad42618f40946d11352890"
format_stamp: "Formatted at 2026-06-19 18:38:30 on dist-test-slave-rxv6"
server_key: "f77f47709bb5d6da16bae250d86232be"
server_key_iv: "df5d47ce8532b7c90fd2a76ea6986b96"
server_key_version: "encryptionkey@0"
I20260619 18:38:30.954994 10621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:30.981709 10621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:30.985641 10621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:30.985858 10621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:30.994057 10621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38865
I20260619 18:38:30.994369 10688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38865 every 8 connection(s)
I20260619 18:38:30.995287 10621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:38:30.998828 10689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:31.002611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 10621
I20260619 18:38:31.002746 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:38:31.003077 392 external_mini_cluster.cc:1428] Setting key dd556d5ab19ffcf03c90c87af2481894
I20260619 18:38:31.006230 10689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890: Bootstrap starting.
I20260619 18:38:31.009361 10689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:31.010476 10689 log.cc:824] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:31.013316 10689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890: No bootstrap required, opened a new log
Jun 19 18:38:31 dist-test-slave-rxv6 krb5kdc[10608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:38:31.017416 10689 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1acc47331ad42618f40946d11352890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } }
I20260619 18:38:31.017715 10689 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:31.017791 10689 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1acc47331ad42618f40946d11352890, State: Initialized, Role: FOLLOWER
I20260619 18:38:31.018464 10689 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1acc47331ad42618f40946d11352890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } }
I20260619 18:38:31.018657 10689 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:38:31.018739 10689 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:38:31.018851 10689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:31.019551 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.004880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.0.98.62:38865)
0619 18:38:31.005376 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:31.005396 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:31.005443 (+ 47us) server_negotiation.cc:407] Connection header received
0619 18:38:31.006299 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:31.006330 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:31.006780 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:31.007141 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:31.008143 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.009343 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.010572 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.011244 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.011453 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.014388 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:31.014394 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:31.014412 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:31.014444 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:31.016212 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:31.016653 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:31.016659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:31.016664 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:31.016788 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:31.017045 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:31.017048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:31.017051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:31.017558 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:31.017771 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:31.018082 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 18:38:31.018458 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":146,"threads_started":1}
I20260619 18:38:31.020009 10689 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1acc47331ad42618f40946d11352890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } }
I20260619 18:38:31.020404 10689 leader_election.cc:302] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1acc47331ad42618f40946d11352890; no voters:
I20260619 18:38:31.020750 10689 leader_election.cc:288] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:38:31.020991 10694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:38:31.021284 10694 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [term 1 LEADER]: Becoming Leader. State: Replica: d1acc47331ad42618f40946d11352890, State: Running, Role: LEADER
I20260619 18:38:31.021696 10694 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1acc47331ad42618f40946d11352890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } }
I20260619 18:38:31.022370 10689 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:38:31.023779 10695 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1acc47331ad42618f40946d11352890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1acc47331ad42618f40946d11352890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } } }
I20260619 18:38:31.023813 10696 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1acc47331ad42618f40946d11352890. Latest consensus state: current_term: 1 leader_uuid: "d1acc47331ad42618f40946d11352890" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1acc47331ad42618f40946d11352890" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38865 } } }
I20260619 18:38:31.023906 10695 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:31.023919 10696 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890 [sys.catalog]: This master's current role is: LEADER
I20260619 18:38:31.024353 10703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:38:31.028182 10703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:38:31.035419 10703 catalog_manager.cc:1347] Generated new cluster ID: e811a2136ef64095b2df593f5fcc313d
I20260619 18:38:31.035542 10703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:38:31.056344 10703 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:38:31.057981 10703 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:38:31.069434 10703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890: Generated new TSK 0
I20260619 18:38:31.070653 10703 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:38:31.106287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:31.248826 10715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:31.249404 10715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:31.249460 10715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:31.254050 10715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:31.254140 10715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:31.254213 10715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:38:31.261314 10715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:31.262858 10715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:31.264148 10715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:38:31.275902 10715 server_base.cc:1034] running on GCE node
W20260619 18:38:31.275851 10720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:31.275852 10721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:31.275852 10723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:31.276680 10715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:31.277561 10715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:31.278728 10715 hybrid_clock.cc:648] HybridClock initialized: now 1781894311278689 us; error 58 us; skew 500 ppm
Jun 19 18:38:31 dist-test-slave-rxv6 krb5kdc[10608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:31.282119 10715 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:38:31.283324 10715 webserver.cc:460] Webserver started at http://127.0.98.1:41917/ using document root <none> and password file <none>
I20260619 18:38:31.284005 10715 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:31.284077 10715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:31.284333 10715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:31.286926 10715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "51c7b0675e174fb7ab90d0b0fd1ef2d9"
format_stamp: "Formatted at 2026-06-19 18:38:31 on dist-test-slave-rxv6"
server_key: "8e9e3757be172f23abeb9bbadbd68f20"
server_key_iv: "6fa339507cca12cb5cd2784698400b5b"
server_key_version: "encryptionkey@0"
I20260619 18:38:31.287705 10715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "51c7b0675e174fb7ab90d0b0fd1ef2d9"
format_stamp: "Formatted at 2026-06-19 18:38:31 on dist-test-slave-rxv6"
server_key: "8e9e3757be172f23abeb9bbadbd68f20"
server_key_iv: "6fa339507cca12cb5cd2784698400b5b"
server_key_version: "encryptionkey@0"
I20260619 18:38:31.292474 10715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:38:31.295405 10730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:31.296904 10715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:38:31.297031 10715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "51c7b0675e174fb7ab90d0b0fd1ef2d9"
format_stamp: "Formatted at 2026-06-19 18:38:31 on dist-test-slave-rxv6"
server_key: "8e9e3757be172f23abeb9bbadbd68f20"
server_key_iv: "6fa339507cca12cb5cd2784698400b5b"
server_key_version: "encryptionkey@0"
I20260619 18:38:31.297152 10715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:31.314667 10715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:31.318231 10715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:31.318468 10715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:31.319254 10715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:31.320313 10715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:31.320363 10715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:31.320423 10715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:31.320456 10715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:31.331459 10715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33777
I20260619 18:38:31.331478 10843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33777 every 8 connection(s)
I20260619 18:38:31.332675 10715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:38:31.335390 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 10715
I20260619 18:38:31.335553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:38:31.335870 392 external_mini_cluster.cc:1428] Setting key a4b41d7d943d050981c1b190f1fca50a
Jun 19 18:38:31 dist-test-slave-rxv6 krb5kdc[10608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 18:38:31.350068 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.335131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42801 (local address 127.0.98.62:38865)
0619 18:38:31.335351 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:31.335356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:31.336465 (+ 1109us) server_negotiation.cc:407] Connection header received
0619 18:38:31.338112 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:31.338116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:31.338176 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:31.338266 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:31.339754 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.340363 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.341682 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.342029 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.342139 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.345628 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:31.345632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:31.345634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:31.345664 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:31.347499 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:31.348101 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:31.348104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:31.348106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:31.348212 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:31.348607 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:31.348611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:31.348613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:31.348889 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:31.348999 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:31.349635 (+ 636us) server_negotiation.cc:299] Negotiation successful
0619 18:38:31.349819 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 18:38:31.351318 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.335668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38865 (local address 127.0.98.1:42801)
0619 18:38:31.336266 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:31.336340 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:31.337827 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:31.338461 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:31.338470 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:31.338938 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:31.339572 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:31.339590 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.340518 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.340522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:31.341543 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:31.341554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.342200 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.342205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:31.342404 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.343273 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:31.343301 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:31.345371 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:31.347628 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:31.347635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:31.347654 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:31.347978 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:31.348303 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:31.348306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:31.348309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:31.348511 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:31.349020 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:31.349026 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:31.349375 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 18:38:31.349578 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 18:38:31.349983 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":171,"threads_started":1}
I20260619 18:38:31.352903 10844 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38865
I20260619 18:38:31.353240 10844 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:31.354055 10844 heartbeater.cc:511] Master 127.0.98.62:38865 requested a full tablet report, sending...
I20260619 18:38:31.356132 10652 ts_manager.cc:194] Registered new tserver with Master: 51c7b0675e174fb7ab90d0b0fd1ef2d9 (127.0.98.1:33777)
I20260619 18:38:31.357496 10652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:38:31.365095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:31.494015 10849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:31.494311 10849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:31.494362 10849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:31.498370 10849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:31.498422 10849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:31.498479 10849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:38:31.503968 10849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:31.505234 10849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:31.506479 10849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:31.517659 10855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:31.517635 10854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:31.517635 10857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:31.518029 10849 server_base.cc:1034] running on GCE node
I20260619 18:38:31.518502 10849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:31.519320 10849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:31.520504 10849 hybrid_clock.cc:648] HybridClock initialized: now 1781894311520475 us; error 40 us; skew 500 ppm
Jun 19 18:38:31 dist-test-slave-rxv6 krb5kdc[10608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:31.523871 10849 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:38:31.524969 10849 webserver.cc:460] Webserver started at http://127.0.98.2:36127/ using document root <none> and password file <none>
I20260619 18:38:31.525617 10849 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:31.525687 10849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:31.525962 10849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:31.528468 10849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "5392ecd2b5cf49bf992ccf4d18b17f2a"
format_stamp: "Formatted at 2026-06-19 18:38:31 on dist-test-slave-rxv6"
server_key: "25ce70ec296d336736b042df1acf9cfa"
server_key_iv: "818fa98126b5c7fa65c6ad2390705fa3"
server_key_version: "encryptionkey@0"
I20260619 18:38:31.529094 10849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5392ecd2b5cf49bf992ccf4d18b17f2a"
format_stamp: "Formatted at 2026-06-19 18:38:31 on dist-test-slave-rxv6"
server_key: "25ce70ec296d336736b042df1acf9cfa"
server_key_iv: "818fa98126b5c7fa65c6ad2390705fa3"
server_key_version: "encryptionkey@0"
I20260619 18:38:31.533442 10849 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 18:38:31.536289 10864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:31.537873 10849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:38:31.538053 10849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "5392ecd2b5cf49bf992ccf4d18b17f2a"
format_stamp: "Formatted at 2026-06-19 18:38:31 on dist-test-slave-rxv6"
server_key: "25ce70ec296d336736b042df1acf9cfa"
server_key_iv: "818fa98126b5c7fa65c6ad2390705fa3"
server_key_version: "encryptionkey@0"
I20260619 18:38:31.538167 10849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:31.568159 10849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:31.571959 10849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:31.572136 10849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:31.572799 10849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:31.573809 10849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:31.573863 10849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:31.573937 10849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:31.573968 10849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:31.583636 10849 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44195
I20260619 18:38:31.583694 10977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44195 every 8 connection(s)
I20260619 18:38:31.584748 10849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:38:31.593518 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 10849
I20260619 18:38:31.593636 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:38:31.593995 392 external_mini_cluster.cc:1428] Setting key 0fe45ac60347194d1c9a68f530e5b6d0
Jun 19 18:38:31 dist-test-slave-rxv6 krb5kdc[10608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:38:31.599990 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.586981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33433 (local address 127.0.98.62:38865)
0619 18:38:31.587142 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:31.587146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:31.588173 (+ 1027us) server_negotiation.cc:407] Connection header received
0619 18:38:31.589655 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:31.589659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:31.589727 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:31.589804 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:31.591400 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.591987 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.593239 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.593472 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.593567 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.596492 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:31.596495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:31.596497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:31.596524 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:31.597881 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:31.598394 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:31.598398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:31.598400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:31.598478 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:31.598812 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:31.598815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:31.598817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:31.599043 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:31.599137 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:31.599662 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 18:38:31.599805 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:38:31.601107 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.587445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38865 (local address 127.0.98.2:33433)
0619 18:38:31.587988 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:31.588050 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:31.589418 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:31.590059 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:31.590068 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:31.590503 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:31.591211 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:31.591227 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.592115 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.592118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:31.593109 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:31.593117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.593625 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.593629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:31.593826 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.594581 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:31.594604 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:31.596324 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:31.598006 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:31.598012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:31.598032 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:31.598303 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:31.598575 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:31.598578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:31.598579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:31.598724 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:31.599131 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:31.599136 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:31.599425 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 18:38:31.599602 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 18:38:31.599909 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":153,"threads_started":1}
I20260619 18:38:31.602432 10978 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38865
I20260619 18:38:31.602725 10978 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:31.603309 10978 heartbeater.cc:511] Master 127.0.98.62:38865 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 18:38:31.604523 10652 ts_manager.cc:194] Registered new tserver with Master: 5392ecd2b5cf49bf992ccf4d18b17f2a (127.0.98.2:44195)
I20260619 18:38:31.605206 10652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:38:31.624866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:38:31.749926 10983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:38:31.750247 10983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:38:31.750295 10983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:38:31.754489 10983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:38:31.754565 10983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:38:31.754631 10983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:38:31.760480 10983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:38:31.761830 10983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:38:31.763049 10983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:38:31.774055 10991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:31.774055 10988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:38:31.774075 10989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:38:31.774152 10983 server_base.cc:1034] running on GCE node
I20260619 18:38:31.774784 10983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:38:31.775542 10983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:38:31.776706 10983 hybrid_clock.cc:648] HybridClock initialized: now 1781894311776684 us; error 33 us; skew 500 ppm
Jun 19 18:38:31 dist-test-slave-rxv6 krb5kdc[10608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:38:31.779839 10983 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:38:31.780884 10983 webserver.cc:460] Webserver started at http://127.0.98.3:42801/ using document root <none> and password file <none>
I20260619 18:38:31.781564 10983 fs_manager.cc:359] Metadata directory not provided
I20260619 18:38:31.781632 10983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:38:31.781833 10983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:38:31.784317 10983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "2b1ba2c8202c421ea6894b6d887e68a7"
format_stamp: "Formatted at 2026-06-19 18:38:31 on dist-test-slave-rxv6"
server_key: "cc4251ff4210016985caeaddac108e95"
server_key_iv: "e4d0c5b4412e9b95b60e0b3388e23730"
server_key_version: "encryptionkey@0"
I20260619 18:38:31.784922 10983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2b1ba2c8202c421ea6894b6d887e68a7"
format_stamp: "Formatted at 2026-06-19 18:38:31 on dist-test-slave-rxv6"
server_key: "cc4251ff4210016985caeaddac108e95"
server_key_iv: "e4d0c5b4412e9b95b60e0b3388e23730"
server_key_version: "encryptionkey@0"
I20260619 18:38:31.789280 10983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:38:31.792270 10998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:31.794127 10983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:38:31.794256 10983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "2b1ba2c8202c421ea6894b6d887e68a7"
format_stamp: "Formatted at 2026-06-19 18:38:31 on dist-test-slave-rxv6"
server_key: "cc4251ff4210016985caeaddac108e95"
server_key_iv: "e4d0c5b4412e9b95b60e0b3388e23730"
server_key_version: "encryptionkey@0"
I20260619 18:38:31.794386 10983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:38:31.816586 10983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:38:31.820076 10983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:38:31.820258 10983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:38:31.820910 10983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:38:31.821967 10983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:38:31.822026 10983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:31.822079 10983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:38:31.822108 10983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:38:31.832037 10983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34563
I20260619 18:38:31.832060 11111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34563 every 8 connection(s)
I20260619 18:38:31.833101 10983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:38:31.842711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 10983
I20260619 18:38:31.842854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:38:31.843187 392 external_mini_cluster.cc:1428] Setting key e6687bd5683a2b43afe0c0f7863aa4bf
Jun 19 18:38:31 dist-test-slave-rxv6 krb5kdc[10608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:38:31.851498 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.835320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51495 (local address 127.0.98.62:38865)
0619 18:38:31.835475 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:31.835479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:31.836456 (+ 977us) server_negotiation.cc:407] Connection header received
0619 18:38:31.837955 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:31.837959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:31.838061 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:31.838141 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:31.839535 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.840119 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.841340 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.841571 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.841666 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.847932 (+ 6266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:31.847935 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:31.847938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:31.847966 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:31.849239 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:31.849781 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:31.849785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:31.849786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:31.849873 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:31.850254 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:31.850257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:31.850259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:31.850520 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:31.850601 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:31.851208 (+ 607us) server_negotiation.cc:299] Negotiation successful
0619 18:38:31.851349 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:38:31.852548 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.835767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38865 (local address 127.0.98.3:51495)
0619 18:38:31.836285 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:31.836332 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:31.837697 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:31.838357 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:31.838366 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:31.838786 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:38:31.839380 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:31.839397 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.840245 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.840248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:31.841218 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:31.841228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.841727 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.841731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:31.841944 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.846025 (+ 4081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:38:31.846044 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:38:31.847761 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:38:31.849363 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:31.849370 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:31.849386 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:31.849670 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:31.850012 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:38:31.850015 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:38:31.850016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:38:31.850167 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:38:31.850619 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:38:31.850626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:38:31.850933 (+ 307us) client_negotiation.cc:769] Sending connection context
0619 18:38:31.851125 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 18:38:31.851457 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":134,"threads_started":1}
I20260619 18:38:31.853950 11112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38865
I20260619 18:38:31.854223 11112 heartbeater.cc:464] Registering TS with master...
I20260619 18:38:31.854782 11112 heartbeater.cc:511] Master 127.0.98.62:38865 requested a full tablet report, sending...
I20260619 18:38:31.855960 10652 ts_manager.cc:194] Registered new tserver with Master: 2b1ba2c8202c421ea6894b6d887e68a7 (127.0.98.3:34563)
I20260619 18:38:31.856666 10652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51495
I20260619 18:38:31.857239 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:38:31.867607 10692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.859780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39344 (local address 127.0.98.62:38865)
0619 18:38:31.859910 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:31.859915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:31.860131 (+ 216us) server_negotiation.cc:407] Connection header received
0619 18:38:31.860269 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:31.860273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:31.860320 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:31.860393 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:38:31.861137 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.861691 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.862610 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.862837 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.862933 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.864453 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:38:31.864457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:38:31.864459 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:38:31.864486 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:38:31.865778 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:31.866276 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:31.866280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:31.866282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:31.866367 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:38:31.866618 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:38:31.866621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:38:31.866623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:38:31.866887 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:38:31.867060 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:31.867277 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 18:38:31.867410 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:38:31.871047 10653 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:38:31.874653 10653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:38:31.890728 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.884813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33777 (local address 127.0.0.1:51504)
0619 18:38:31.885895 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:31.885982 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:31.886142 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:31.886689 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:31.886695 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:31.886717 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:31.886868 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:31.886875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.888010 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.888013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:31.889205 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:31.889214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.890120 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.890124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:31.890229 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.890272 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 18:38:31.890391 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 18:38:31.890538 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"spinlock_wait_cycles":7296,"thread_start_us":102,"threads_started":1}
I20260619 18:38:31.890734 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.885611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.0.98.1:33777)
0619 18:38:31.886054 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:31.886060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:31.886086 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:38:31.886308 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:31.886315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:31.886554 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:31.886695 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:31.887006 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.887873 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.889337 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.890010 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.890118 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.890289 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:31.890435 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 18:38:31.890558 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":121,"threads_started":1}
I20260619 18:38:31.891450 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.884551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34563 (local address 127.0.0.1:59570)
0619 18:38:31.885896 (+ 1345us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:31.885990 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:31.886143 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:31.886898 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:31.886901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:31.886918 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:31.887010 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:31.887016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.888491 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.888495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:31.889818 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:31.889828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.891021 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.891025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:31.891121 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.891140 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 18:38:31.891207 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 18:38:31.891286 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"mutex_wait_us":41,"spinlock_wait_cycles":239360,"thread_start_us":127,"threads_started":1}
I20260619 18:38:31.891484 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.884999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.0.98.3:34563)
0619 18:38:31.885528 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:31.885542 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:31.886127 (+ 585us) server_negotiation.cc:407] Connection header received
0619 18:38:31.886311 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:31.886315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:31.886555 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:31.886709 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:31.887117 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.888332 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.889967 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.890783 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.890900 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.891046 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:31.891245 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 18:38:31.891345 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":159,"threads_started":1}
I20260619 18:38:31.892187 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.884502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44195 (local address 127.0.0.1:38514)
0619 18:38:31.886313 (+ 1811us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:31.886324 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:31.886408 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:31.886854 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:31.886857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:31.886871 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:31.886966 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:31.886972 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.888499 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.888502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:31.889820 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:31.889828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.891794 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.891798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:31.891899 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.891924 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:38:31.891996 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 18:38:31.892060 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":114,"thread_start_us":117,"threads_started":1}
I20260619 18:38:31.892813 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.885502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38514 (local address 127.0.98.2:44195)
0619 18:38:31.886054 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:31.886067 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:31.886418 (+ 351us) server_negotiation.cc:407] Connection header received
0619 18:38:31.886497 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:31.886501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:31.886656 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:31.886798 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:31.887094 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.888332 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.890009 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.891257 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.891365 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.892547 (+ 1182us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:31.892604 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 18:38:31.892694 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":156,"threads_started":1}
I20260619 18:38:31.894853 11046 tablet_service.cc:1467] Processing CreateTablet for tablet 801aba1f9d2f4f17ab17f768730e2e40 (DEFAULT_TABLE table=test-table [id=a802bec05fde4c2c9eb241e17090467f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:31.894846 10778 tablet_service.cc:1467] Processing CreateTablet for tablet 801aba1f9d2f4f17ab17f768730e2e40 (DEFAULT_TABLE table=test-table [id=a802bec05fde4c2c9eb241e17090467f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:31.895228 10912 tablet_service.cc:1467] Processing CreateTablet for tablet 801aba1f9d2f4f17ab17f768730e2e40 (DEFAULT_TABLE table=test-table [id=a802bec05fde4c2c9eb241e17090467f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:38:31.896198 10778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 801aba1f9d2f4f17ab17f768730e2e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:31.896199 11046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 801aba1f9d2f4f17ab17f768730e2e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:31.896235 10912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 801aba1f9d2f4f17ab17f768730e2e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:38:31.903887 11129 tablet_bootstrap.cc:492] T 801aba1f9d2f4f17ab17f768730e2e40 P 51c7b0675e174fb7ab90d0b0fd1ef2d9: Bootstrap starting.
I20260619 18:38:31.905390 11131 tablet_bootstrap.cc:492] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7: Bootstrap starting.
I20260619 18:38:31.905390 11130 tablet_bootstrap.cc:492] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a: Bootstrap starting.
I20260619 18:38:31.906847 11129 tablet_bootstrap.cc:654] T 801aba1f9d2f4f17ab17f768730e2e40 P 51c7b0675e174fb7ab90d0b0fd1ef2d9: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:31.907907 11129 log.cc:824] T 801aba1f9d2f4f17ab17f768730e2e40 P 51c7b0675e174fb7ab90d0b0fd1ef2d9: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:31.908016 11131 tablet_bootstrap.cc:654] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:31.908517 11130 tablet_bootstrap.cc:654] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a: Neither blocks nor log segments found. Creating new log.
I20260619 18:38:31.909430 11131 log.cc:824] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:31.909430 11130 log.cc:824] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:38:31.910614 11129 tablet_bootstrap.cc:492] T 801aba1f9d2f4f17ab17f768730e2e40 P 51c7b0675e174fb7ab90d0b0fd1ef2d9: No bootstrap required, opened a new log
I20260619 18:38:31.910871 11129 ts_tablet_manager.cc:1397] T 801aba1f9d2f4f17ab17f768730e2e40 P 51c7b0675e174fb7ab90d0b0fd1ef2d9: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 18:38:31.911998 11131 tablet_bootstrap.cc:492] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7: No bootstrap required, opened a new log
I20260619 18:38:31.912005 11130 tablet_bootstrap.cc:492] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a: No bootstrap required, opened a new log
I20260619 18:38:31.912212 11130 ts_tablet_manager.cc:1397] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260619 18:38:31.912212 11131 ts_tablet_manager.cc:1397] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260619 18:38:31.915545 11129 raft_consensus.cc:348] T 801aba1f9d2f4f17ab17f768730e2e40 P 51c7b0675e174fb7ab90d0b0fd1ef2d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5392ecd2b5cf49bf992ccf4d18b17f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44195 } } peers { permanent_uuid: "2b1ba2c8202c421ea6894b6d887e68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34563 } } peers { permanent_uuid: "51c7b0675e174fb7ab90d0b0fd1ef2d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33777 } }
I20260619 18:38:31.915869 11129 raft_consensus.cc:374] T 801aba1f9d2f4f17ab17f768730e2e40 P 51c7b0675e174fb7ab90d0b0fd1ef2d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:31.915932 11129 raft_consensus.cc:729] T 801aba1f9d2f4f17ab17f768730e2e40 P 51c7b0675e174fb7ab90d0b0fd1ef2d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c7b0675e174fb7ab90d0b0fd1ef2d9, State: Initialized, Role: FOLLOWER
I20260619 18:38:31.916479 11129 consensus_queue.cc:260] T 801aba1f9d2f4f17ab17f768730e2e40 P 51c7b0675e174fb7ab90d0b0fd1ef2d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5392ecd2b5cf49bf992ccf4d18b17f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44195 } } peers { permanent_uuid: "2b1ba2c8202c421ea6894b6d887e68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34563 } } peers { permanent_uuid: "51c7b0675e174fb7ab90d0b0fd1ef2d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33777 } }
I20260619 18:38:31.916512 11131 raft_consensus.cc:348] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5392ecd2b5cf49bf992ccf4d18b17f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44195 } } peers { permanent_uuid: "2b1ba2c8202c421ea6894b6d887e68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34563 } } peers { permanent_uuid: "51c7b0675e174fb7ab90d0b0fd1ef2d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33777 } }
I20260619 18:38:31.916507 11130 raft_consensus.cc:348] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5392ecd2b5cf49bf992ccf4d18b17f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44195 } } peers { permanent_uuid: "2b1ba2c8202c421ea6894b6d887e68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34563 } } peers { permanent_uuid: "51c7b0675e174fb7ab90d0b0fd1ef2d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33777 } }
I20260619 18:38:31.916821 11131 raft_consensus.cc:374] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:31.916821 11130 raft_consensus.cc:374] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:38:31.916878 11131 raft_consensus.cc:729] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b1ba2c8202c421ea6894b6d887e68a7, State: Initialized, Role: FOLLOWER
I20260619 18:38:31.916878 11130 raft_consensus.cc:729] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5392ecd2b5cf49bf992ccf4d18b17f2a, State: Initialized, Role: FOLLOWER
I20260619 18:38:31.917397 11130 consensus_queue.cc:260] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5392ecd2b5cf49bf992ccf4d18b17f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44195 } } peers { permanent_uuid: "2b1ba2c8202c421ea6894b6d887e68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34563 } } peers { permanent_uuid: "51c7b0675e174fb7ab90d0b0fd1ef2d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33777 } }
I20260619 18:38:31.917395 11131 consensus_queue.cc:260] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5392ecd2b5cf49bf992ccf4d18b17f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44195 } } peers { permanent_uuid: "2b1ba2c8202c421ea6894b6d887e68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34563 } } peers { permanent_uuid: "51c7b0675e174fb7ab90d0b0fd1ef2d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33777 } }
I20260619 18:38:31.917482 10844 heartbeater.cc:503] Master 127.0.98.62:38865 was elected leader, sending a full tablet report...
I20260619 18:38:31.917893 11129 ts_tablet_manager.cc:1428] T 801aba1f9d2f4f17ab17f768730e2e40 P 51c7b0675e174fb7ab90d0b0fd1ef2d9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 18:38:31.918545 11130 ts_tablet_manager.cc:1428] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260619 18:38:31.918545 11131 ts_tablet_manager.cc:1428] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7: Time spent starting tablet: real 0.006s user 0.002s sys 0.006s
I20260619 18:38:31.918937 10978 heartbeater.cc:503] Master 127.0.98.62:38865 was elected leader, sending a full tablet report...
I20260619 18:38:31.918937 11112 heartbeater.cc:503] Master 127.0.98.62:38865 was elected leader, sending a full tablet report...
I20260619 18:38:31.948704 11137 raft_consensus.cc:482] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:38:31.948961 11137 raft_consensus.cc:504] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5392ecd2b5cf49bf992ccf4d18b17f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44195 } } peers { permanent_uuid: "2b1ba2c8202c421ea6894b6d887e68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34563 } } peers { permanent_uuid: "51c7b0675e174fb7ab90d0b0fd1ef2d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33777 } }
I20260619 18:38:31.950474 11137 leader_election.cc:288] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5392ecd2b5cf49bf992ccf4d18b17f2a (127.0.98.2:44195), 51c7b0675e174fb7ab90d0b0fd1ef2d9 (127.0.98.1:33777)
I20260619 18:38:31.954282 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.950958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60771 (local address 127.0.98.2:44195)
0619 18:38:31.951099 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:31.951104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:31.951120 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:38:31.951186 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:31.951189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:31.951242 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:31.951309 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:31.951591 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.952135 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.953257 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.953679 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.953775 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.953851 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:31.954053 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 18:38:31.954139 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:38:31.954282 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.950710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44195 (local address 127.0.98.3:60771)
0619 18:38:31.950973 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:31.950991 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:31.951097 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:31.951332 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:31.951335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:31.951361 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:31.951487 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:31.951493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.952238 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.952242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:31.953142 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:31.953150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.953793 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.953796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:31.953936 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.953963 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 18:38:31.954034 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 18:38:31.954120 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":28,"spinlock_wait_cycles":3456}
I20260619 18:38:31.954422 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.950913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51119 (local address 127.0.98.1:33777)
0619 18:38:31.951065 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:31.951070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:31.951337 (+ 267us) server_negotiation.cc:407] Connection header received
0619 18:38:31.951456 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:31.951459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:31.951509 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:31.951575 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:38:31.951822 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.952362 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.953436 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.953870 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.953981 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.954055 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:31.954211 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 18:38:31.954292 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:38:31.954456 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.950744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33777 (local address 127.0.98.3:51119)
0619 18:38:31.951238 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:31.951248 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 18:38:31.951351 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:38:31.951591 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:38:31.951594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:38:31.951608 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:38:31.951729 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:31.951735 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.952462 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.952465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:31.953322 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:38:31.953330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.953996 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.954000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:38:31.954103 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.954121 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 18:38:31.954193 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 18:38:31.954260 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":57,"thread_start_us":168,"threads_started":1}
I20260619 18:38:31.955111 10798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801aba1f9d2f4f17ab17f768730e2e40" candidate_uuid: "2b1ba2c8202c421ea6894b6d887e68a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c7b0675e174fb7ab90d0b0fd1ef2d9" is_pre_election: true
I20260619 18:38:31.955114 10932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801aba1f9d2f4f17ab17f768730e2e40" candidate_uuid: "2b1ba2c8202c421ea6894b6d887e68a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5392ecd2b5cf49bf992ccf4d18b17f2a" is_pre_election: true
I20260619 18:38:31.955458 10798 raft_consensus.cc:2454] T 801aba1f9d2f4f17ab17f768730e2e40 P 51c7b0675e174fb7ab90d0b0fd1ef2d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ba2c8202c421ea6894b6d887e68a7 in term 0.
I20260619 18:38:31.955456 10932 raft_consensus.cc:2454] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b1ba2c8202c421ea6894b6d887e68a7 in term 0.
I20260619 18:38:31.955945 10999 leader_election.cc:302] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ba2c8202c421ea6894b6d887e68a7, 51c7b0675e174fb7ab90d0b0fd1ef2d9; no voters:
I20260619 18:38:31.956251 11137 raft_consensus.cc:2781] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:38:31.956341 11137 raft_consensus.cc:482] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:38:31.956423 11137 raft_consensus.cc:3037] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:31.957664 11137 raft_consensus.cc:504] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5392ecd2b5cf49bf992ccf4d18b17f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44195 } } peers { permanent_uuid: "2b1ba2c8202c421ea6894b6d887e68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34563 } } peers { permanent_uuid: "51c7b0675e174fb7ab90d0b0fd1ef2d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33777 } }
I20260619 18:38:31.958081 11137 leader_election.cc:288] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [CANDIDATE]: Term 1 election: Requested vote from peers 5392ecd2b5cf49bf992ccf4d18b17f2a (127.0.98.2:44195), 51c7b0675e174fb7ab90d0b0fd1ef2d9 (127.0.98.1:33777)
I20260619 18:38:31.958451 10798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801aba1f9d2f4f17ab17f768730e2e40" candidate_uuid: "2b1ba2c8202c421ea6894b6d887e68a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c7b0675e174fb7ab90d0b0fd1ef2d9"
I20260619 18:38:31.958588 10798 raft_consensus.cc:3037] T 801aba1f9d2f4f17ab17f768730e2e40 P 51c7b0675e174fb7ab90d0b0fd1ef2d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:31.959909 10798 raft_consensus.cc:2454] T 801aba1f9d2f4f17ab17f768730e2e40 P 51c7b0675e174fb7ab90d0b0fd1ef2d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1ba2c8202c421ea6894b6d887e68a7 in term 1.
I20260619 18:38:31.960234 10999 leader_election.cc:302] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b1ba2c8202c421ea6894b6d887e68a7, 51c7b0675e174fb7ab90d0b0fd1ef2d9; no voters:
I20260619 18:38:31.960307 10932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "801aba1f9d2f4f17ab17f768730e2e40" candidate_uuid: "2b1ba2c8202c421ea6894b6d887e68a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5392ecd2b5cf49bf992ccf4d18b17f2a"
I20260619 18:38:31.960465 11137 raft_consensus.cc:2781] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:38:31.960448 10932 raft_consensus.cc:3037] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:38:31.960727 11137 raft_consensus.cc:686] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [term 1 LEADER]: Becoming Leader. State: Replica: 2b1ba2c8202c421ea6894b6d887e68a7, State: Running, Role: LEADER
I20260619 18:38:31.961134 11137 consensus_queue.cc:237] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5392ecd2b5cf49bf992ccf4d18b17f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44195 } } peers { permanent_uuid: "2b1ba2c8202c421ea6894b6d887e68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34563 } } peers { permanent_uuid: "51c7b0675e174fb7ab90d0b0fd1ef2d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33777 } }
I20260619 18:38:31.961664 10932 raft_consensus.cc:2454] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b1ba2c8202c421ea6894b6d887e68a7 in term 1.
I20260619 18:38:31.964993 10651 catalog_manager.cc:5495] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b1ba2c8202c421ea6894b6d887e68a7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2b1ba2c8202c421ea6894b6d887e68a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5392ecd2b5cf49bf992ccf4d18b17f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b1ba2c8202c421ea6894b6d887e68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51c7b0675e174fb7ab90d0b0fd1ef2d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33777 } health_report { overall_health: UNKNOWN } } }
I20260619 18:38:31.992901 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:31.989155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.0.98.3:34563)
0619 18:38:31.989296 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 18:38:31.989300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:38:31.989313 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:38:31.989423 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:38:31.989426 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:38:31.989512 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:38:31.989651 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:38:31.989930 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.990475 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.991321 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:38:31.991529 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:38:31.991597 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:38:31.991929 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:38:31.992490 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:38:31.992583 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 18:38:31.992665 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 18:38:31.992730 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:38:32.000640 10798 raft_consensus.cc:1261] T 801aba1f9d2f4f17ab17f768730e2e40 P 51c7b0675e174fb7ab90d0b0fd1ef2d9 [term 1 FOLLOWER]: Refusing update from remote peer 2b1ba2c8202c421ea6894b6d887e68a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:38:32.000640 10932 raft_consensus.cc:1261] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a [term 1 FOLLOWER]: Refusing update from remote peer 2b1ba2c8202c421ea6894b6d887e68a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:38:32.001600 11139 consensus_queue.cc:1035] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51c7b0675e174fb7ab90d0b0fd1ef2d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:38:32.001807 11137 consensus_queue.cc:1035] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5392ecd2b5cf49bf992ccf4d18b17f2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:38:32.011323 11142 mvcc.cc:205] Tried to move back new op lower bound from 7298639101945118720 to 7298639101796433920. Current Snapshot: MvccSnapshot[applied={T|T < 7298639101945118720}]
I20260619 18:38:32.012396 11143 mvcc.cc:205] Tried to move back new op lower bound from 7298639101945118720 to 7298639101796433920. Current Snapshot: MvccSnapshot[applied={T|T < 7298639101945118720}]
I20260619 18:38:32.023041 10651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:38:32.023313 10651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:38:32.026391 10651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1acc47331ad42618f40946d11352890: Sending DeleteTablet for 3 replicas of tablet 801aba1f9d2f4f17ab17f768730e2e40
I20260619 18:38:32.027393 10912 tablet_service.cc:1514] Processing DeleteTablet for tablet 801aba1f9d2f4f17ab17f768730e2e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38514
I20260619 18:38:32.027644 11046 tablet_service.cc:1514] Processing DeleteTablet for tablet 801aba1f9d2f4f17ab17f768730e2e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59570
I20260619 18:38:32.027787 10778 tablet_service.cc:1514] Processing DeleteTablet for tablet 801aba1f9d2f4f17ab17f768730e2e40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:38:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51504
I20260619 18:38:32.028283 11153 tablet_replica.cc:330] T 801aba1f9d2f4f17ab17f768730e2e40 P 51c7b0675e174fb7ab90d0b0fd1ef2d9: stopping tablet replica
I20260619 18:38:32.028285 11152 tablet_replica.cc:330] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7: stopping tablet replica
I20260619 18:38:32.028716 11152 raft_consensus.cc:2229] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:38:32.028716 11153 raft_consensus.cc:2229] T 801aba1f9d2f4f17ab17f768730e2e40 P 51c7b0675e174fb7ab90d0b0fd1ef2d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:38:32.029006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 10715
I20260619 18:38:32.030086 11152 raft_consensus.cc:2258] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:32.030292 11151 tablet_replica.cc:330] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a: stopping tablet replica
I20260619 18:38:32.030661 11151 raft_consensus.cc:2229] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:38:32.031035 11151 raft_consensus.cc:2258] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:38:32.032218 11152 ts_tablet_manager.cc:1905] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:32.033576 11151 ts_tablet_manager.cc:1905] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:38:32.035878 11152 ts_tablet_manager.cc:1918] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:38:32.036020 11152 log.cc:1196] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/801aba1f9d2f4f17ab17f768730e2e40
I20260619 18:38:32.036381 11152 ts_tablet_manager.cc:1939] T 801aba1f9d2f4f17ab17f768730e2e40 P 2b1ba2c8202c421ea6894b6d887e68a7: Deleting consensus metadata
I20260619 18:38:32.037002 11151 ts_tablet_manager.cc:1918] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:38:32.037191 11151 log.cc:1196] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/801aba1f9d2f4f17ab17f768730e2e40
I20260619 18:38:32.037569 11151 ts_tablet_manager.cc:1939] T 801aba1f9d2f4f17ab17f768730e2e40 P 5392ecd2b5cf49bf992ccf4d18b17f2a: Deleting consensus metadata
W20260619 18:38:32.037611 10637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33777 (error 108)
I20260619 18:38:32.037997 10638 catalog_manager.cc:4841] TS 2b1ba2c8202c421ea6894b6d887e68a7 (127.0.98.3:34563): tablet 801aba1f9d2f4f17ab17f768730e2e40 (table test-table [id=a802bec05fde4c2c9eb241e17090467f]) successfully deleted
I20260619 18:38:32.038918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 10849
I20260619 18:38:32.039690 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:32.038968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33777 (local address 127.0.0.1:51514)
0619 18:38:32.039517 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:32.039595 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438}
W20260619 18:38:32.039861 10637 catalog_manager.cc:4584] TS 51c7b0675e174fb7ab90d0b0fd1ef2d9 (127.0.98.1:33777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 801aba1f9d2f4f17ab17f768730e2e40: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33777: connect: Connection refused (error 111)
I20260619 18:38:32.048887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 10983
I20260619 18:38:32.049077 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:38:32.048743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44195 (local address 127.0.0.1:38524)
0619 18:38:32.048890 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 18:38:32.048956 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 18:38:32.055004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 10621
2026-06-19T18:38:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2366 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:38:32 dist-test-slave-rxv6 krb5kdc[11157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:38:32 dist-test-slave-rxv6 krb5kdc[11157](info): set up 2 sockets
Jun 19 18:38:32 dist-test-slave-rxv6 krb5kdc[11157](info): commencing operation
krb5kdc: starting...
W20260619 18:38:33.019022 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:38:33 dist-test-slave-rxv6 krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:38:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:38:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 18:38:33.200896 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskqHyw_j/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/postgres -l logfile start
2026-06-19 18:38:35.731 UTC [11187] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 18:38:35.731 UTC [11187] LOG: listening on IPv4 address "127.0.98.20", port 33477
2026-06-19 18:38:35.734 UTC [11187] LOG: listening on Unix socket "/tmp/.s.PGSQL.33477"
2026-06-19 18:38:35.757 UTC [11190] LOG: database system was shut down at 2026-06-19 18:38:34 UTC
2026-06-19 18:38:35.761 UTC [11187] LOG: database system is ready to accept connections
I20260619 18:38:36.634943 392 mini_postgres.cc:96] Postgres bound to 33477
2026-06-19 18:38:36.642 UTC [11198] FATAL: database "slave" does not exist
127.0.98.20:33477 - accepting connections
I20260619 18:38:36.643203 392 mini_ranger.cc:161] Starting Ranger...
I20260619 18:38:36.659909 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 18:38:37.174336 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 18:38:37.174432 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ranger-admin
2026-06-19 18:38:37,457 [I] DB FLAVOR :POSTGRES
2026-06-19 18:38:37,458 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:38:37,458 [I] Checking connection..
2026-06-19 18:38:37,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 18:38:37,744 [I] Checking connection passed.
2026-06-19 18:38:37,745 [I] --------- Verifying version history table ---------
2026-06-19 18:38:37,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:38:38,028 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:38:38,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:38:38,311 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:38:38,312 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 18:38:38,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 18:38:38.559 UTC [11262] WARNING: there is no transaction in progress
2026-06-19 18:38:38,565 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 18:38:38,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:38:38,845 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 18:38:38,845 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:38:38,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 18:38:39,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 18:38:39,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rxv6.c.gcp-upstream.internal','N') ;"
2026-06-19 18:38:39,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:38:39,893 [I] Table x_portal_user does not exist in database ranger
2026-06-19 18:38:39,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:38:40,159 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 18:38:40,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:38:40,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:38:40,695 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 18:38:40,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 18:38:41.741 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 18:38:41.758 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 18:38:41.775 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 18:38:41.790 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 18:38:41.808 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 18:38:41.827 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 18:38:41.893 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 18:38:41.900 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 18:38:41.914 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 18:38:41.925 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 18:38:41.935 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 18:38:41.950 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 18:38:41.960 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 18:38:41.967 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 18:38:41.974 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 18:38:41.981 UTC [11370] WARNING: there is no transaction in progress
2026-06-19 18:38:42,666 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 18:38:42,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 18:38:42,935 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 18:38:42,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:38:43,181 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 18:38:43,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:38:43,446 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 18:38:43,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:38:43,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:38:43,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rxv6.c.gcp-upstream.internal';"
2026-06-19 18:38:44,241 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 18:38:44,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 18:38:44,488 [I] DB_PATCHES have already been applied
I20260619 18:38:44.495397 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskqHyw_j/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 18:38:44.495486 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 18:38:44.499677 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:34963
Jun 19, 2026 6:38:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:38:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:38:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:38:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:38:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:38:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:38:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 18:38:45.526181 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.883s user 0.004s sys 0.008s
Jun 19, 2026 6:39:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:39:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 6:39:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:39:06 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:39:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:39:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:39:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:39:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:39:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:39:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:39:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:39:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:39:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:39:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:39:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:39:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:39:06 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:39:06 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 18:39:07.546691 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 18:39:07 dist-test-slave-rxv6 krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894347, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 18:39:07.639354 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 18:39:07.648485 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 18:39:08.535817 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 18:39:08.535962 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ranger-kms
I20260619 18:39:08.536056 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:33477
2026-06-19 18:39:08,707 [I] DB FLAVOR :POSTGRES
2026-06-19 18:39:08,707 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:39:08,707 [I] Checking connection
2026-06-19 18:39:08,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 18:39:08,973 [I] connection success
2026-06-19 18:39:08,973 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 18:39:08,973 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 18:39:08,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 18:39:09,202 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 18:39:09,202 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:39:09,202 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 18:39:09,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33477/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 18:39:09,460 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 18:39:09.933653 392 mini_ranger_kms.cc:310] Created kms service
I20260619 18:39:10.126120 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 18:39:10.175010 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 18:39:10 dist-test-slave-rxv6 krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894347, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 18:39:10.575263 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 18:39:10.575409 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 18:39:10.575433 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 18:39:10.579003 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11550
I20260619 18:39:10.579138 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:47365
Jun 19, 2026 6:39:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:39:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:39:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:39:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:39:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:39:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 18:39:12.629802 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.990s user 0.015s sys 0.141s
I20260619 18:39:12.629941 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 18:39:12.630041 392 mini_ranger_kms.cc:401] 127.0.98.20:47365/kms/v1/keys
Jun 19 18:39:12 dist-test-slave-rxv6 krb5kdc[11157](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894352, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 18:39:13 dist-test-slave-rxv6 krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894352, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 18:39:13 dist-test-slave-rxv6 krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894352, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 18:39:15 dist-test-slave-rxv6 krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:39:15.745270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41409
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41409
--ranger_config_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47365/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:39:15.889864 11601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:39:15.890144 11601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:39:15.890187 11601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:39:15.894186 11601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:39:15.894245 11601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:39:15.894263 11601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:39:15.894279 11601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:39:15.894296 11601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:39:15.899703 11601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:47365/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41409
--ranger_config_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:39:15.900759 11601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:39:15.901793 11601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:39:15.910833 11606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:39:15.910756 11609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:39:15.910897 11607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:39:15.910777 11601 server_base.cc:1034] running on GCE node
I20260619 18:39:15.911679 11601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:39:15.912904 11601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:39:15.914067 11601 hybrid_clock.cc:648] HybridClock initialized: now 1781894355914044 us; error 41 us; skew 500 ppm
Jun 19 18:39:15 dist-test-slave-rxv6 krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:39:15.916882 11601 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:39:15.917928 11601 webserver.cc:460] Webserver started at http://127.0.98.62:43503/ using document root <none> and password file <none>
I20260619 18:39:15.918566 11601 fs_manager.cc:359] Metadata directory not provided
I20260619 18:39:15.918637 11601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:39:15.918849 11601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:39:15 dist-test-slave-rxv6 krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 18:39:15.975399 11601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "b56558a3ba4a4edb8a2473221915f2a0"
format_stamp: "Formatted at 2026-06-19 18:39:15 on dist-test-slave-rxv6"
server_key: "92e3702e9ced15e60e4520a5adda8e00"
server_key_iv: "c73c6b783517cc24ab2d228723fa773f"
server_key_version: "kuduclusterkey@0"
I20260619 18:39:15.976014 11601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "b56558a3ba4a4edb8a2473221915f2a0"
format_stamp: "Formatted at 2026-06-19 18:39:15 on dist-test-slave-rxv6"
server_key: "92e3702e9ced15e60e4520a5adda8e00"
server_key_iv: "c73c6b783517cc24ab2d228723fa773f"
server_key_version: "kuduclusterkey@0"
I20260619 18:39:15.980031 11601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:39:15.994797 11616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:39:15.996191 11601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:39:15.996333 11601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "b56558a3ba4a4edb8a2473221915f2a0"
format_stamp: "Formatted at 2026-06-19 18:39:15 on dist-test-slave-rxv6"
server_key: "92e3702e9ced15e60e4520a5adda8e00"
server_key_iv: "c73c6b783517cc24ab2d228723fa773f"
server_key_version: "kuduclusterkey@0"
I20260619 18:39:15.996454 11601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:39:16.017256 11601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:39:16.020627 11601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:39:16.020804 11601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:39:16.028488 11601 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41409
I20260619 18:39:16.028538 11668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41409 every 8 connection(s)
I20260619 18:39:16.029482 11601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:39:16.032389 11669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:39:16.033365 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 11601
I20260619 18:39:16.033486 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:39:16.033825 392 external_mini_cluster.cc:1428] Setting key b8c95a04b6c73fcc246f0a8f87f0a42a
I20260619 18:39:16.038914 11669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0: Bootstrap starting.
I20260619 18:39:16.041393 11669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0: Neither blocks nor log segments found. Creating new log.
I20260619 18:39:16.042225 11669 log.cc:824] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:39:16 dist-test-slave-rxv6 krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:39:16.044513 11669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0: No bootstrap required, opened a new log
I20260619 18:39:16.047802 11669 raft_consensus.cc:348] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56558a3ba4a4edb8a2473221915f2a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41409 } }
I20260619 18:39:16.048030 11669 raft_consensus.cc:374] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:39:16.048082 11669 raft_consensus.cc:729] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b56558a3ba4a4edb8a2473221915f2a0, State: Initialized, Role: FOLLOWER
I20260619 18:39:16.048600 11669 consensus_queue.cc:260] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56558a3ba4a4edb8a2473221915f2a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41409 } }
I20260619 18:39:16.048751 11669 raft_consensus.cc:388] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:39:16.048820 11669 raft_consensus.cc:482] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:39:16.048916 11669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:39:16.049427 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:16.035349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57464 (local address 127.0.98.62:41409)
0619 18:39:16.035832 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 18:39:16.035852 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:39:16.035898 (+ 46us) server_negotiation.cc:407] Connection header received
0619 18:39:16.036763 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:39:16.036788 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:39:16.037244 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:39:16.037647 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:39:16.038493 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:16.039987 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:16.041153 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:16.041885 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:16.042077 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:16.044534 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:39:16.044540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:39:16.044553 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:39:16.044579 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:39:16.046737 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:39:16.047146 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:39:16.047151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:39:16.047156 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:39:16.047257 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:39:16.047506 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:39:16.047509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:39:16.047511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:39:16.047889 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:39:16.048074 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 18:39:16.048319 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 18:39:16.048602 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":139,"threads_started":1}
I20260619 18:39:16.049854 11669 raft_consensus.cc:504] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56558a3ba4a4edb8a2473221915f2a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41409 } }
I20260619 18:39:16.050220 11669 leader_election.cc:302] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b56558a3ba4a4edb8a2473221915f2a0; no voters:
I20260619 18:39:16.050554 11669 leader_election.cc:288] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:39:16.050637 11674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:39:16.050838 11674 raft_consensus.cc:686] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [term 1 LEADER]: Becoming Leader. State: Replica: b56558a3ba4a4edb8a2473221915f2a0, State: Running, Role: LEADER
I20260619 18:39:16.051221 11674 consensus_queue.cc:237] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56558a3ba4a4edb8a2473221915f2a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41409 } }
I20260619 18:39:16.051829 11669 sys_catalog.cc:564] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:39:16.052353 11674 sys_catalog.cc:455] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b56558a3ba4a4edb8a2473221915f2a0. Latest consensus state: current_term: 1 leader_uuid: "b56558a3ba4a4edb8a2473221915f2a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56558a3ba4a4edb8a2473221915f2a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41409 } } }
I20260619 18:39:16.052541 11674 sys_catalog.cc:458] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [sys.catalog]: This master's current role is: LEADER
I20260619 18:39:16.052853 11675 sys_catalog.cc:455] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b56558a3ba4a4edb8a2473221915f2a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b56558a3ba4a4edb8a2473221915f2a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41409 } } }
I20260619 18:39:16.053028 11675 sys_catalog.cc:458] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0 [sys.catalog]: This master's current role is: LEADER
I20260619 18:39:16.052970 11681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:39:16.055980 11681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:39:16.056412 11669 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 18:39:16.067776 11681 catalog_manager.cc:1347] Generated new cluster ID: b017f18d3fcb4e90a673182a1af6501a
I20260619 18:39:16.067910 11681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:39:16.078867 11681 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:39:16.080196 11681 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:39:16.091405 11681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0: Generated new TSK 0
I20260619 18:39:16.092324 11681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:39:16.801694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47365/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41409
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:39:16.919734 11714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:39:16.920073 11714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:39:16.920138 11714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:39:16.924865 11714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:39:16.924950 11714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:39:16.925040 11714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:39:16.932964 11714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:47365/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:39:16.934219 11714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:39:16.935242 11714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:39:16.947276 11722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:39:16.947449 11721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:39:16.947279 11724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:39:16.947840 11714 server_base.cc:1034] running on GCE node
I20260619 18:39:16.948447 11714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:39:16.949184 11714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:39:16.950382 11714 hybrid_clock.cc:648] HybridClock initialized: now 1781894356950368 us; error 30 us; skew 500 ppm
Jun 19 18:39:16 dist-test-slave-rxv6 krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:39:16.953655 11714 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:39:16.954633 11714 webserver.cc:460] Webserver started at http://127.0.98.1:39697/ using document root <none> and password file <none>
I20260619 18:39:16.955168 11714 fs_manager.cc:359] Metadata directory not provided
I20260619 18:39:16.955231 11714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:39:16.955425 11714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:39:16 dist-test-slave-rxv6 krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 18:39:16.972307 11714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "f49ea8ff339b42e3b4c7b90cafb932eb"
format_stamp: "Formatted at 2026-06-19 18:39:16 on dist-test-slave-rxv6"
server_key: "834943462e8734f640cd1b3f17cfd24d"
server_key_iv: "1a01d79dda4128876f86f7c06eab9208"
server_key_version: "kuduclusterkey@0"
I20260619 18:39:16.973210 11714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f49ea8ff339b42e3b4c7b90cafb932eb"
format_stamp: "Formatted at 2026-06-19 18:39:16 on dist-test-slave-rxv6"
server_key: "834943462e8734f640cd1b3f17cfd24d"
server_key_iv: "1a01d79dda4128876f86f7c06eab9208"
server_key_version: "kuduclusterkey@0"
I20260619 18:39:16.977551 11714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:39:16.995378 11731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:39:16.996955 11714 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:39:16.997095 11714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "f49ea8ff339b42e3b4c7b90cafb932eb"
format_stamp: "Formatted at 2026-06-19 18:39:16 on dist-test-slave-rxv6"
server_key: "834943462e8734f640cd1b3f17cfd24d"
server_key_iv: "1a01d79dda4128876f86f7c06eab9208"
server_key_version: "kuduclusterkey@0"
I20260619 18:39:16.997243 11714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:39:17.058686 11714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:39:17.062382 11714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:39:17.062577 11714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:39:17.063288 11714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:39:17.064368 11714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:39:17.064428 11714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:39:17.064487 11714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:39:17.064558 11714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:39:17.077333 11714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40329
I20260619 18:39:17.077585 11846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40329 every 8 connection(s)
I20260619 18:39:17.078603 11714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:39:17.080011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 11714
I20260619 18:39:17.080137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:39:17.080440 392 external_mini_cluster.cc:1428] Setting key a963696c04ad1edc6ae731153de5f867
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Jun 19 18:39:17 dist-test-slave-rxv6 krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 18:39:17.098186 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:17.080771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34557 (local address 127.0.98.62:41409)
0619 18:39:17.081086 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0619 18:39:17.081095 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:39:17.082089 (+ 994us) server_negotiation.cc:407] Connection header received
0619 18:39:17.085193 (+ 3104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:39:17.085196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:39:17.085254 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:39:17.085315 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:39:17.087052 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.087919 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.089744 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.090144 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.090225 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:17.093548 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:39:17.093553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:39:17.093556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:39:17.093586 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:39:17.095445 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:39:17.096136 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:39:17.096140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:39:17.096142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:39:17.096245 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:39:17.096693 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:39:17.096697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:39:17.096699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:39:17.096965 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:39:17.097042 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 18:39:17.097822 (+ 780us) server_negotiation.cc:299] Negotiation successful
0619 18:39:17.097995 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":110,"threads_started":1}
I20260619 18:39:17.100553 11850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:17.081291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41409 (local address 127.0.98.1:34557)
0619 18:39:17.081875 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 18:39:17.081962 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0619 18:39:17.083085 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:39:17.085753 (+ 2668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:39:17.085763 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:39:17.086174 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:39:17.086856 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:39:17.086876 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.088100 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.088103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:39:17.089591 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:39:17.089602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.090453 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.090457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:39:17.090674 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:17.091376 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:39:17.091398 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:39:17.093331 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:39:17.095602 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:39:17.095609 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:39:17.095629 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:39:17.095995 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:39:17.096369 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:39:17.096372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:39:17.096374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:39:17.096581 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:39:17.097129 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:39:17.097136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:39:17.097518 (+ 382us) client_negotiation.cc:769] Sending connection context
0619 18:39:17.098517 (+ 999us) client_negotiation.cc:240] Negotiation successful
0619 18:39:17.098885 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":154,"threads_started":1}
I20260619 18:39:17.102169 11847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41409
I20260619 18:39:17.102458 11847 heartbeater.cc:464] Registering TS with master...
I20260619 18:39:17.103109 11847 heartbeater.cc:511] Master 127.0.98.62:41409 requested a full tablet report, sending...
I20260619 18:39:17.105314 11633 ts_manager.cc:194] Registered new tserver with Master: f49ea8ff339b42e3b4c7b90cafb932eb (127.0.98.1:40329)
I20260619 18:39:17.107025 11633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:39:17.114593 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47365/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41409
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:39:17 dist-test-slave-rxv6 krb5kdc[11157](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 18:39:17.275066 11853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:39:17.275458 11853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:39:17.275532 11853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:39:17.281574 11853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:39:17.281677 11853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:39:17.281770 11853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:39:17.289779 11853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:47365/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:39:17.291224 11853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:39:17.292618 11853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:39:17.307579 11859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:39:17.308086 11860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:39:17.308341 11862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:39:17.309065 11853 server_base.cc:1034] running on GCE node
I20260619 18:39:17.309691 11853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:39:17.310595 11853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:39:17.311777 11853 hybrid_clock.cc:648] HybridClock initialized: now 1781894357311744 us; error 52 us; skew 500 ppm
Jun 19 18:39:17 dist-test-slave-rxv6 krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:39:17.314459 11853 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:39:17.315397 11853 webserver.cc:460] Webserver started at http://127.0.98.2:43453/ using document root <none> and password file <none>
I20260619 18:39:17.315934 11853 fs_manager.cc:359] Metadata directory not provided
I20260619 18:39:17.315997 11853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:39:17.316190 11853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:39:17 dist-test-slave-rxv6 krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 18:39:17.333253 11853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "86ffddb1069e413b8c3ff596d20e8832"
format_stamp: "Formatted at 2026-06-19 18:39:17 on dist-test-slave-rxv6"
server_key: "fba928ca2244594e114830eb054d0353"
server_key_iv: "793b78db03d1b5cb9674850861e5b48e"
server_key_version: "kuduclusterkey@0"
I20260619 18:39:17.334039 11853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "86ffddb1069e413b8c3ff596d20e8832"
format_stamp: "Formatted at 2026-06-19 18:39:17 on dist-test-slave-rxv6"
server_key: "fba928ca2244594e114830eb054d0353"
server_key_iv: "793b78db03d1b5cb9674850861e5b48e"
server_key_version: "kuduclusterkey@0"
I20260619 18:39:17.343560 11853 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.003s sys 0.000s
I20260619 18:39:17.374493 11869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:39:17.376412 11853 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260619 18:39:17.376633 11853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "86ffddb1069e413b8c3ff596d20e8832"
format_stamp: "Formatted at 2026-06-19 18:39:17 on dist-test-slave-rxv6"
server_key: "fba928ca2244594e114830eb054d0353"
server_key_iv: "793b78db03d1b5cb9674850861e5b48e"
server_key_version: "kuduclusterkey@0"
I20260619 18:39:17.376938 11853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:39:17.414997 11853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:39:17.418795 11853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:39:17.418991 11853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:39:17.419672 11853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:39:17.420796 11853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:39:17.420855 11853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:39:17.420910 11853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:39:17.420931 11853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:39:17.443874 11853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42891
I20260619 18:39:17.445062 11853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:39:17.446738 11982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42891 every 8 connection(s)
I20260619 18:39:17.455303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 11853
I20260619 18:39:17.455437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:39:17.455757 392 external_mini_cluster.cc:1428] Setting key d18302e0086e73643b621ac12f672979
Jun 19 18:39:17 dist-test-slave-rxv6 krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 18:39:17.475637 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:17.450298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50645 (local address 127.0.98.62:41409)
0619 18:39:17.450419 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 18:39:17.450423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:39:17.450439 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:39:17.451054 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:39:17.451058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:39:17.451114 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:39:17.451164 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:39:17.453040 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.453964 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.455642 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.456021 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.456099 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:17.469704 (+ 13605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:39:17.469709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:39:17.469712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:39:17.469756 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:39:17.471539 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:39:17.473588 (+ 2049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:39:17.473593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:39:17.473595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:39:17.473693 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:39:17.474199 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:39:17.474203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:39:17.474205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:39:17.474451 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:39:17.474527 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 18:39:17.475306 (+ 779us) server_negotiation.cc:299] Negotiation successful
0619 18:39:17.475449 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:39:17.477622 11985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:17.449208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41409 (local address 127.0.98.2:50645)
0619 18:39:17.449713 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 18:39:17.449777 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 18:39:17.450796 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:39:17.451632 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:39:17.451642 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:39:17.452007 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:39:17.452855 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:39:17.452884 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.454165 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.454168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:39:17.455500 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:39:17.455512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.458262 (+ 2750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.458267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:39:17.458480 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:17.460492 (+ 2012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:39:17.460513 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:39:17.463213 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:39:17.472943 (+ 9730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:39:17.472951 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:39:17.472984 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:39:17.473381 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:39:17.473814 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:39:17.473817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:39:17.473819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:39:17.474076 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:39:17.474622 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:39:17.474630 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:39:17.475000 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 18:39:17.476138 (+ 1138us) client_negotiation.cc:240] Negotiation successful
0619 18:39:17.476516 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":122,"threads_started":1}
I20260619 18:39:17.479349 11983 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41409
I20260619 18:39:17.479616 11983 heartbeater.cc:464] Registering TS with master...
I20260619 18:39:17.480255 11983 heartbeater.cc:511] Master 127.0.98.62:41409 requested a full tablet report, sending...
I20260619 18:39:17.481499 11633 ts_manager.cc:194] Registered new tserver with Master: 86ffddb1069e413b8c3ff596d20e8832 (127.0.98.2:42891)
I20260619 18:39:17.482416 11633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:39:17.495159 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:47365/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41409
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:39:17 dist-test-slave-rxv6 krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 18:39:17.657250 11988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:39:17.657603 11988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:39:17.657661 11988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:39:17.663689 11988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:39:17.663838 11988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:39:17.663951 11988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:39:17.671609 11988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:47365/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:39:17.673110 11988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:39:17.674468 11988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:39:17.686733 11994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:39:17.687472 11993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:39:17.688237 11988 server_base.cc:1034] running on GCE node
W20260619 18:39:17.690056 11996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:39:17.690559 11988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:39:17.691284 11988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:39:17.692438 11988 hybrid_clock.cc:648] HybridClock initialized: now 1781894357692380 us; error 73 us; skew 500 ppm
Jun 19 18:39:17 dist-test-slave-rxv6 krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:39:17.695930 11988 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:39:17.697150 11988 webserver.cc:460] Webserver started at http://127.0.98.3:42973/ using document root <none> and password file <none>
I20260619 18:39:17.697856 11988 fs_manager.cc:359] Metadata directory not provided
I20260619 18:39:17.698004 11988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:39:17.698283 11988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:39:17 dist-test-slave-rxv6 krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 18:39:17.712759 11988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "8564dab57f144c6095ca47f5d551c0e9"
format_stamp: "Formatted at 2026-06-19 18:39:17 on dist-test-slave-rxv6"
server_key: "3ccceabc2a7998dbbc4a274cc9cd8684"
server_key_iv: "c22f08ff0ce8483e5877d96943462af9"
server_key_version: "kuduclusterkey@0"
I20260619 18:39:17.713392 11988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8564dab57f144c6095ca47f5d551c0e9"
format_stamp: "Formatted at 2026-06-19 18:39:17 on dist-test-slave-rxv6"
server_key: "3ccceabc2a7998dbbc4a274cc9cd8684"
server_key_iv: "c22f08ff0ce8483e5877d96943462af9"
server_key_version: "kuduclusterkey@0"
I20260619 18:39:17.718415 11988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 18:39:17.734491 12003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:39:17.736114 11988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:39:17.736249 11988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "8564dab57f144c6095ca47f5d551c0e9"
format_stamp: "Formatted at 2026-06-19 18:39:17 on dist-test-slave-rxv6"
server_key: "3ccceabc2a7998dbbc4a274cc9cd8684"
server_key_iv: "c22f08ff0ce8483e5877d96943462af9"
server_key_version: "kuduclusterkey@0"
I20260619 18:39:17.736392 11988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:39:17.765053 11988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:39:17.767937 11988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:39:17.768111 11988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:39:17.768694 11988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:39:17.769554 11988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:39:17.769603 11988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:39:17.769657 11988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:39:17.769682 11988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:39:17.779542 11988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37647
I20260619 18:39:17.779574 12116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37647 every 8 connection(s)
I20260619 18:39:17.780455 11988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:39:17.787622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 11988
I20260619 18:39:17.787757 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:39:17.788058 392 external_mini_cluster.cc:1428] Setting key 16e6c0960053b2f196600d66e3e7acae
Jun 19 18:39:17 dist-test-slave-rxv6 krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:39:17.795336 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:17.782387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57443 (local address 127.0.98.62:41409)
0619 18:39:17.782569 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 18:39:17.782573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:39:17.783430 (+ 857us) server_negotiation.cc:407] Connection header received
0619 18:39:17.784504 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:39:17.784507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:39:17.784564 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:39:17.784637 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:39:17.785666 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.786375 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.787530 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.787846 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.787925 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:17.791411 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:39:17.791414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:39:17.791416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:39:17.791439 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:39:17.792807 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:39:17.793430 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:39:17.793434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:39:17.793436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:39:17.793524 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:39:17.793970 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:39:17.793973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:39:17.793975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:39:17.794229 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:39:17.794304 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:39:17.795023 (+ 719us) server_negotiation.cc:299] Negotiation successful
0619 18:39:17.795152 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 18:39:17.797086 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:17.782797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41409 (local address 127.0.98.3:57443)
0619 18:39:17.783277 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 18:39:17.783328 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:39:17.784299 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:39:17.784802 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:39:17.784809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:39:17.785062 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:39:17.785527 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:39:17.785540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.786496 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.786499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:39:17.787389 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:39:17.787401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.788118 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.788122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:39:17.788323 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:17.789285 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:39:17.789809 (+ 524us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:39:17.791258 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:39:17.792941 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:39:17.792947 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:39:17.792963 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:39:17.793310 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:39:17.793635 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:39:17.793638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:39:17.793641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:39:17.793829 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:39:17.794388 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:39:17.794394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:39:17.794731 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 18:39:17.795728 (+ 997us) client_negotiation.cc:240] Negotiation successful
0619 18:39:17.796075 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":124,"threads_started":1}
I20260619 18:39:17.799420 12117 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41409
I20260619 18:39:17.800273 12117 heartbeater.cc:464] Registering TS with master...
I20260619 18:39:17.800926 12117 heartbeater.cc:511] Master 127.0.98.62:41409 requested a full tablet report, sending...
I20260619 18:39:17.803470 11633 ts_manager.cc:194] Registered new tserver with Master: 8564dab57f144c6095ca47f5d551c0e9 (127.0.98.3:37647)
I20260619 18:39:17.804088 11633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57443
I20260619 18:39:17.814982 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:39:17.824272 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:17.817111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.0.98.62:41409)
0619 18:39:17.817232 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 18:39:17.817237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:39:17.817456 (+ 219us) server_negotiation.cc:407] Connection header received
0619 18:39:17.817564 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:39:17.817567 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:39:17.817623 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:39:17.817686 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:39:17.818410 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.818925 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.819733 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.819951 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.820035 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:17.821324 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:39:17.821328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:39:17.821331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:39:17.821358 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:39:17.822796 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:39:17.823183 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:39:17.823187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:39:17.823189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:39:17.823265 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:39:17.823500 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:39:17.823503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:39:17.823504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:39:17.823687 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:39:17.823802 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 18:39:17.824014 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 18:39:17.824120 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:39:17.827319 11633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:39:17.830711 11633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:39:17.852623 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:17.840312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42891 (local address 127.0.0.1:60080)
0619 18:39:17.846789 (+ 6477us) negotiation.cc:107] Waiting for socket to connect
0619 18:39:17.846804 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:39:17.846901 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:39:17.847731 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:39:17.847738 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:39:17.847762 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:39:17.847900 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:39:17.847909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.849302 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.849306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:39:17.850627 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:39:17.850637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.851959 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.851963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:39:17.852076 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:17.852125 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 18:39:17.852253 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 18:39:17.852380 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6409,"thread_start_us":90,"threads_started":1}
I20260619 18:39:17.853632 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:17.840676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60080 (local address 127.0.98.2:42891)
0619 18:39:17.841430 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0619 18:39:17.841442 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:39:17.847010 (+ 5568us) server_negotiation.cc:407] Connection header received
0619 18:39:17.847102 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:39:17.847108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:39:17.847359 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:39:17.847522 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:39:17.848039 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.849185 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.850782 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.851656 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.851754 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:17.851902 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 18:39:17.853231 (+ 1329us) server_negotiation.cc:299] Negotiation successful
0619 18:39:17.853427 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":85,"threads_started":1}
I20260619 18:39:17.856501 11917 tablet_service.cc:1467] Processing CreateTablet for tablet 730adf088f3641a1a02e5a24c0e7af36 (DEFAULT_TABLE table=test-table [id=2d85cf390b3340ccaa71686cdbf52349]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:39:17.857447 11917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 730adf088f3641a1a02e5a24c0e7af36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:39:17.865002 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:17.839270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37647 (local address 127.0.0.1:54456)
0619 18:39:17.840929 (+ 1659us) negotiation.cc:107] Waiting for socket to connect
0619 18:39:17.840978 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:39:17.841112 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:39:17.851000 (+ 9888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:39:17.851003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:39:17.851020 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:39:17.851115 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:39:17.851121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.853369 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.853372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:39:17.862114 (+ 8742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:39:17.862126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.864583 (+ 2457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.864587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:39:17.864699 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:17.864727 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 18:39:17.864777 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 18:39:17.864856 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1480,"thread_start_us":95,"threads_started":1}
I20260619 18:39:17.866024 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:17.840079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54456 (local address 127.0.98.3:37647)
0619 18:39:17.849512 (+ 9433us) server_negotiation.cc:206] Beginning negotiation
0619 18:39:17.849523 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:39:17.849547 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:39:17.849631 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:39:17.849637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:39:17.850688 (+ 1051us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:39:17.850851 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:39:17.851235 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.852275 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.862321 (+ 10046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.863163 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.865517 (+ 2354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:17.865681 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 18:39:17.865737 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 18:39:17.865834 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9314,"thread_start_us":138,"threads_started":1}
I20260619 18:39:17.867503 12135 tablet_bootstrap.cc:492] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832: Bootstrap starting.
I20260619 18:39:17.868782 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:17.839345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40329 (local address 127.0.0.1:35442)
0619 18:39:17.841145 (+ 1800us) negotiation.cc:107] Waiting for socket to connect
0619 18:39:17.841157 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:39:17.841225 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:39:17.853142 (+ 11917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:39:17.853145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:39:17.853161 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:39:17.853241 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:39:17.853247 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.865227 (+ 11980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.865231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:39:17.866365 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:39:17.866378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.868354 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.868358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:39:17.868448 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:17.868473 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:39:17.868539 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 18:39:17.868627 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1741,"thread_start_us":74,"threads_started":1}
I20260619 18:39:17.869714 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:17.839541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35442 (local address 127.0.98.1:40329)
0619 18:39:17.852537 (+ 12996us) server_negotiation.cc:206] Beginning negotiation
0619 18:39:17.852544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:39:17.852567 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:39:17.852650 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:39:17.852656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:39:17.852856 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:39:17.852995 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:39:17.863316 (+ 10321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.864405 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.866515 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:17.867296 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:17.867396 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:17.869271 (+ 1875us) server_negotiation.cc:1035] Waiting for connection context
0619 18:39:17.869326 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 18:39:17.869568 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12881,"thread_start_us":88,"threads_started":1}
I20260619 18:39:17.869360 12033 tablet_service.cc:1467] Processing CreateTablet for tablet 730adf088f3641a1a02e5a24c0e7af36 (DEFAULT_TABLE table=test-table [id=2d85cf390b3340ccaa71686cdbf52349]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:39:17.870154 12135 tablet_bootstrap.cc:654] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832: Neither blocks nor log segments found. Creating new log.
I20260619 18:39:17.870513 12033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 730adf088f3641a1a02e5a24c0e7af36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:39:17.871434 12135 log.cc:824] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832: Log is configured to *not* fsync() on all Append() calls
I20260619 18:39:17.872727 11780 tablet_service.cc:1467] Processing CreateTablet for tablet 730adf088f3641a1a02e5a24c0e7af36 (DEFAULT_TABLE table=test-table [id=2d85cf390b3340ccaa71686cdbf52349]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:39:17.873517 12135 tablet_bootstrap.cc:492] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832: No bootstrap required, opened a new log
I20260619 18:39:17.873696 12135 ts_tablet_manager.cc:1397] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:39:17.873769 11780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 730adf088f3641a1a02e5a24c0e7af36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:39:17.878249 12135 raft_consensus.cc:348] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8564dab57f144c6095ca47f5d551c0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37647 } } peers { permanent_uuid: "f49ea8ff339b42e3b4c7b90cafb932eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40329 } } peers { permanent_uuid: "86ffddb1069e413b8c3ff596d20e8832" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } }
I20260619 18:39:17.878538 12135 raft_consensus.cc:374] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:39:17.878603 12135 raft_consensus.cc:729] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86ffddb1069e413b8c3ff596d20e8832, State: Initialized, Role: FOLLOWER
I20260619 18:39:17.879076 12135 consensus_queue.cc:260] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8564dab57f144c6095ca47f5d551c0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37647 } } peers { permanent_uuid: "f49ea8ff339b42e3b4c7b90cafb932eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40329 } } peers { permanent_uuid: "86ffddb1069e413b8c3ff596d20e8832" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } }
I20260619 18:39:17.880059 12135 ts_tablet_manager.cc:1428] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 18:39:17.880422 11983 heartbeater.cc:503] Master 127.0.98.62:41409 was elected leader, sending a full tablet report...
I20260619 18:39:17.882711 12139 tablet_bootstrap.cc:492] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb: Bootstrap starting.
I20260619 18:39:17.884991 12137 tablet_bootstrap.cc:492] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9: Bootstrap starting.
I20260619 18:39:17.887109 12139 tablet_bootstrap.cc:654] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb: Neither blocks nor log segments found. Creating new log.
I20260619 18:39:17.887538 12137 tablet_bootstrap.cc:654] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9: Neither blocks nor log segments found. Creating new log.
I20260619 18:39:17.888057 12139 log.cc:824] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb: Log is configured to *not* fsync() on all Append() calls
I20260619 18:39:17.888324 12137 log.cc:824] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9: Log is configured to *not* fsync() on all Append() calls
I20260619 18:39:17.890197 12139 tablet_bootstrap.cc:492] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb: No bootstrap required, opened a new log
I20260619 18:39:17.890197 12137 tablet_bootstrap.cc:492] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9: No bootstrap required, opened a new log
I20260619 18:39:17.890386 12139 ts_tablet_manager.cc:1397] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 18:39:17.890386 12137 ts_tablet_manager.cc:1397] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:39:17.894825 12137 raft_consensus.cc:348] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8564dab57f144c6095ca47f5d551c0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37647 } } peers { permanent_uuid: "f49ea8ff339b42e3b4c7b90cafb932eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40329 } } peers { permanent_uuid: "86ffddb1069e413b8c3ff596d20e8832" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } }
I20260619 18:39:17.895287 12137 raft_consensus.cc:374] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:39:17.895376 12137 raft_consensus.cc:729] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8564dab57f144c6095ca47f5d551c0e9, State: Initialized, Role: FOLLOWER
I20260619 18:39:17.895941 12137 consensus_queue.cc:260] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8564dab57f144c6095ca47f5d551c0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37647 } } peers { permanent_uuid: "f49ea8ff339b42e3b4c7b90cafb932eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40329 } } peers { permanent_uuid: "86ffddb1069e413b8c3ff596d20e8832" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } }
I20260619 18:39:17.896658 12139 raft_consensus.cc:348] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8564dab57f144c6095ca47f5d551c0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37647 } } peers { permanent_uuid: "f49ea8ff339b42e3b4c7b90cafb932eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40329 } } peers { permanent_uuid: "86ffddb1069e413b8c3ff596d20e8832" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } }
I20260619 18:39:17.896972 12139 raft_consensus.cc:374] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:39:17.897059 12139 raft_consensus.cc:729] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49ea8ff339b42e3b4c7b90cafb932eb, State: Initialized, Role: FOLLOWER
I20260619 18:39:17.897086 12137 ts_tablet_manager.cc:1428] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 18:39:17.897444 12117 heartbeater.cc:503] Master 127.0.98.62:41409 was elected leader, sending a full tablet report...
I20260619 18:39:17.897681 12139 consensus_queue.cc:260] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8564dab57f144c6095ca47f5d551c0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37647 } } peers { permanent_uuid: "f49ea8ff339b42e3b4c7b90cafb932eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40329 } } peers { permanent_uuid: "86ffddb1069e413b8c3ff596d20e8832" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } }
I20260619 18:39:17.899009 12139 ts_tablet_manager.cc:1428] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260619 18:39:17.899402 11847 heartbeater.cc:503] Master 127.0.98.62:41409 was elected leader, sending a full tablet report...
Jun 19 18:39:17 dist-test-slave-rxv6 krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 18:39:17.949083 11984 tablet.cc:2369] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:39:18.048439 12118 tablet.cc:2369] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:39:18.080848 11848 tablet.cc:2369] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:39:18.183902 12143 raft_consensus.cc:482] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:39:18.184171 12143 raft_consensus.cc:504] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8564dab57f144c6095ca47f5d551c0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37647 } } peers { permanent_uuid: "f49ea8ff339b42e3b4c7b90cafb932eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40329 } } peers { permanent_uuid: "86ffddb1069e413b8c3ff596d20e8832" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } }
I20260619 18:39:18.185724 12143 leader_election.cc:288] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8564dab57f144c6095ca47f5d551c0e9 (127.0.98.3:37647), 86ffddb1069e413b8c3ff596d20e8832 (127.0.98.2:42891)
I20260619 18:39:18.190130 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:18.186098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59283 (local address 127.0.98.2:42891)
0619 18:39:18.186319 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 18:39:18.186324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:39:18.186696 (+ 372us) server_negotiation.cc:407] Connection header received
0619 18:39:18.186826 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:39:18.186829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:39:18.186871 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:39:18.186933 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:39:18.187345 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:18.187837 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:18.189225 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:18.189627 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:18.189716 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:18.189777 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 18:39:18.189946 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 18:39:18.190014 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260619 18:39:18.190165 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:18.185965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42891 (local address 127.0.98.1:59283)
0619 18:39:18.186574 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 18:39:18.186589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:39:18.186730 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:39:18.187036 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:39:18.187039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:39:18.187052 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:39:18.187223 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:39:18.187229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:39:18.187947 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:39:18.187950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:39:18.189109 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:39:18.189117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:39:18.189740 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:39:18.189744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:39:18.189824 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:18.189843 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 18:39:18.189926 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 18:39:18.189995 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":22,"spinlock_wait_cycles":1408,"thread_start_us":126,"threads_started":1}
I20260619 18:39:18.190735 11924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730adf088f3641a1a02e5a24c0e7af36" candidate_uuid: "f49ea8ff339b42e3b4c7b90cafb932eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ffddb1069e413b8c3ff596d20e8832" is_pre_election: true
I20260619 18:39:18.191020 11924 raft_consensus.cc:2454] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f49ea8ff339b42e3b4c7b90cafb932eb in term 0.
I20260619 18:39:18.191854 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:18.185965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37647 (local address 127.0.98.1:34545)
0619 18:39:18.186544 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0619 18:39:18.186556 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:39:18.186675 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:39:18.187000 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:39:18.187004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:39:18.187025 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:39:18.187217 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:39:18.187224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:39:18.188858 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:39:18.188862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:39:18.190078 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:39:18.190089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:39:18.191403 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:39:18.191406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:39:18.191500 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:18.191522 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:39:18.191599 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 18:39:18.191676 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":292,"spinlock_wait_cycles":35072,"thread_start_us":116,"threads_started":1}
I20260619 18:39:18.191874 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:18.186186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34545 (local address 127.0.98.3:37647)
0619 18:39:18.186317 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 18:39:18.186321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:39:18.186644 (+ 323us) server_negotiation.cc:407] Connection header received
0619 18:39:18.186773 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:39:18.186776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:39:18.186829 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:39:18.186917 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:39:18.187965 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:18.188707 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:18.190543 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:18.191166 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:18.191247 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:18.191307 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 18:39:18.191658 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 18:39:18.191759 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:39:18.191965 11733 leader_election.cc:302] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86ffddb1069e413b8c3ff596d20e8832, f49ea8ff339b42e3b4c7b90cafb932eb; no voters:
I20260619 18:39:18.192250 12143 raft_consensus.cc:2781] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:39:18.192399 12143 raft_consensus.cc:482] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:39:18.192478 12143 raft_consensus.cc:3037] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:39:18.192497 12055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730adf088f3641a1a02e5a24c0e7af36" candidate_uuid: "f49ea8ff339b42e3b4c7b90cafb932eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8564dab57f144c6095ca47f5d551c0e9" is_pre_election: true
I20260619 18:39:18.192766 12055 raft_consensus.cc:2454] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f49ea8ff339b42e3b4c7b90cafb932eb in term 0.
I20260619 18:39:18.193851 12143 raft_consensus.cc:504] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8564dab57f144c6095ca47f5d551c0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37647 } } peers { permanent_uuid: "f49ea8ff339b42e3b4c7b90cafb932eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40329 } } peers { permanent_uuid: "86ffddb1069e413b8c3ff596d20e8832" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } }
I20260619 18:39:18.194298 12143 leader_election.cc:288] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [CANDIDATE]: Term 1 election: Requested vote from peers 8564dab57f144c6095ca47f5d551c0e9 (127.0.98.3:37647), 86ffddb1069e413b8c3ff596d20e8832 (127.0.98.2:42891)
I20260619 18:39:18.194746 12055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730adf088f3641a1a02e5a24c0e7af36" candidate_uuid: "f49ea8ff339b42e3b4c7b90cafb932eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8564dab57f144c6095ca47f5d551c0e9"
I20260619 18:39:18.194875 12055 raft_consensus.cc:3037] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:39:18.194919 11924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730adf088f3641a1a02e5a24c0e7af36" candidate_uuid: "f49ea8ff339b42e3b4c7b90cafb932eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86ffddb1069e413b8c3ff596d20e8832"
I20260619 18:39:18.195035 11924 raft_consensus.cc:3037] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:39:18.196203 12055 raft_consensus.cc:2454] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f49ea8ff339b42e3b4c7b90cafb932eb in term 1.
I20260619 18:39:18.196210 11924 raft_consensus.cc:2454] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f49ea8ff339b42e3b4c7b90cafb932eb in term 1.
I20260619 18:39:18.196540 11734 leader_election.cc:302] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8564dab57f144c6095ca47f5d551c0e9, f49ea8ff339b42e3b4c7b90cafb932eb; no voters:
I20260619 18:39:18.196746 12143 raft_consensus.cc:2781] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:39:18.197007 12143 raft_consensus.cc:686] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [term 1 LEADER]: Becoming Leader. State: Replica: f49ea8ff339b42e3b4c7b90cafb932eb, State: Running, Role: LEADER
I20260619 18:39:18.197386 12143 consensus_queue.cc:237] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8564dab57f144c6095ca47f5d551c0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37647 } } peers { permanent_uuid: "f49ea8ff339b42e3b4c7b90cafb932eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40329 } } peers { permanent_uuid: "86ffddb1069e413b8c3ff596d20e8832" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } }
I20260619 18:39:18.201357 11633 catalog_manager.cc:5495] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb reported cstate change: term changed from 0 to 1, leader changed from <none> to f49ea8ff339b42e3b4c7b90cafb932eb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f49ea8ff339b42e3b4c7b90cafb932eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8564dab57f144c6095ca47f5d551c0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f49ea8ff339b42e3b4c7b90cafb932eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86ffddb1069e413b8c3ff596d20e8832" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 } health_report { overall_health: UNKNOWN } } }
I20260619 18:39:18.217444 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:18.213119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.0.98.1:40329)
0619 18:39:18.213354 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 18:39:18.213359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:39:18.213375 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:39:18.213463 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:39:18.213466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:39:18.213530 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:39:18.213650 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:39:18.213858 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:18.214691 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:18.215598 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:39:18.215949 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:39:18.216062 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:39:18.216397 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:39:18.216998 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:39:18.217130 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 18:39:18.217201 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 18:39:18.217274 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260619 18:39:18.224642 11924 raft_consensus.cc:1261] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832 [term 1 FOLLOWER]: Refusing update from remote peer f49ea8ff339b42e3b4c7b90cafb932eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:39:18.224694 12055 raft_consensus.cc:1261] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9 [term 1 FOLLOWER]: Refusing update from remote peer f49ea8ff339b42e3b4c7b90cafb932eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:39:18.225416 12155 consensus_queue.cc:1035] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "8564dab57f144c6095ca47f5d551c0e9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:39:18.225616 12143 consensus_queue.cc:1035] T 730adf088f3641a1a02e5a24c0e7af36 P f49ea8ff339b42e3b4c7b90cafb932eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "86ffddb1069e413b8c3ff596d20e8832" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:39:18.245184 11633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:39:18.245352 11633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:39:18.247562 11633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b56558a3ba4a4edb8a2473221915f2a0: Sending DeleteTablet for 3 replicas of tablet 730adf088f3641a1a02e5a24c0e7af36
I20260619 18:39:18.248349 12033 tablet_service.cc:1514] Processing DeleteTablet for tablet 730adf088f3641a1a02e5a24c0e7af36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:39:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54456
I20260619 18:39:18.248457 11917 tablet_service.cc:1514] Processing DeleteTablet for tablet 730adf088f3641a1a02e5a24c0e7af36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:39:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60080
I20260619 18:39:18.249027 12169 tablet_replica.cc:330] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832: stopping tablet replica
I20260619 18:39:18.249238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 11714
I20260619 18:39:18.249408 12169 raft_consensus.cc:2229] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:39:18.249893 12169 raft_consensus.cc:2258] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:39:18.250133 12168 tablet_replica.cc:330] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9: stopping tablet replica
I20260619 18:39:18.250425 12168 raft_consensus.cc:2229] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:39:18.250711 12168 raft_consensus.cc:2258] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:39:18.251492 12169 ts_tablet_manager.cc:1905] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:39:18.252447 12168 ts_tablet_manager.cc:1905] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:39:18.254452 12169 ts_tablet_manager.cc:1918] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:39:18.254561 12169 log.cc:1196] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/730adf088f3641a1a02e5a24c0e7af36
I20260619 18:39:18.254856 12169 ts_tablet_manager.cc:1939] T 730adf088f3641a1a02e5a24c0e7af36 P 86ffddb1069e413b8c3ff596d20e8832: Deleting consensus metadata
I20260619 18:39:18.255211 12168 ts_tablet_manager.cc:1918] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:39:18.255400 12168 log.cc:1196] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/730adf088f3641a1a02e5a24c0e7af36
I20260619 18:39:18.255769 12168 ts_tablet_manager.cc:1939] T 730adf088f3641a1a02e5a24c0e7af36 P 8564dab57f144c6095ca47f5d551c0e9: Deleting consensus metadata
I20260619 18:39:18.256294 11618 catalog_manager.cc:4841] TS 86ffddb1069e413b8c3ff596d20e8832 (127.0.98.2:42891): tablet 730adf088f3641a1a02e5a24c0e7af36 (table test-table [id=2d85cf390b3340ccaa71686cdbf52349]) successfully deleted
I20260619 18:39:18.257184 11619 catalog_manager.cc:4841] TS 8564dab57f144c6095ca47f5d551c0e9 (127.0.98.3:37647): tablet 730adf088f3641a1a02e5a24c0e7af36 (table test-table [id=2d85cf390b3340ccaa71686cdbf52349]) successfully deleted
W20260619 18:39:18.258507 11618 connection.cc:535] client connection to 127.0.98.1:40329 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 18:39:18.258567 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 11853
W20260619 18:39:18.258661 11618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 18:39:18.260044 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:39:18.259747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40329 (local address 127.0.0.1:35460)
0619 18:39:18.259881 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 18:39:18.259953 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 18:39:18.260244 11618 catalog_manager.cc:4584] TS f49ea8ff339b42e3b4c7b90cafb932eb (127.0.98.1:40329): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 730adf088f3641a1a02e5a24c0e7af36: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40329: connect: Connection refused (error 111)
I20260619 18:39:18.264122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 11988
I20260619 18:39:18.269029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 11601
I20260619 18:39:18.276712 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 18:39:18.611991 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 18:39:18.612112 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 18:39:18.947178 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 18:39:18.947 UTC [11187] LOG: received smart shutdown request
2026-06-19 18:39:18.950 UTC [11187] LOG: background worker "logical replication launcher" (PID 11196) exited with exit code 1
2026-06-19 18:39:18.950 UTC [11191] LOG: shutting down
2026-06-19 18:39:19.107 UTC [11187] LOG: database system is shut down
2026-06-19T18:39:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47141 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:39:19 dist-test-slave-rxv6 krb5kdc[12179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:39:19 dist-test-slave-rxv6 krb5kdc[12179](info): set up 2 sockets
Jun 19 18:39:19 dist-test-slave-rxv6 krb5kdc[12179](info): commencing operation
krb5kdc: starting...
W20260619 18:39:20.161780 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:39:20 dist-test-slave-rxv6 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:39:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:39:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 18:39:20.324148 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskqHyw_j/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/postgres -l logfile start
2026-06-19 18:39:22.773 UTC [12209] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 18:39:22.773 UTC [12209] LOG: listening on IPv4 address "127.0.98.20", port 58157
2026-06-19 18:39:22.777 UTC [12209] LOG: listening on Unix socket "/tmp/.s.PGSQL.58157"
2026-06-19 18:39:22.803 UTC [12212] LOG: database system was shut down at 2026-06-19 18:39:21 UTC
2026-06-19 18:39:22.807 UTC [12209] LOG: database system is ready to accept connections
I20260619 18:39:23.673601 392 mini_postgres.cc:96] Postgres bound to 58157
2026-06-19 18:39:23.679 UTC [12220] FATAL: database "slave" does not exist
127.0.98.20:58157 - accepting connections
I20260619 18:39:23.679924 392 mini_ranger.cc:161] Starting Ranger...
I20260619 18:39:23.693830 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 18:39:24.158358 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 18:39:24.158428 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-admin
2026-06-19 18:39:24,421 [I] DB FLAVOR :POSTGRES
2026-06-19 18:39:24,422 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:39:24,422 [I] Checking connection..
2026-06-19 18:39:24,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 18:39:24,678 [I] Checking connection passed.
2026-06-19 18:39:24,679 [I] --------- Verifying version history table ---------
2026-06-19 18:39:24,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:39:24,945 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:39:24,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:39:25,199 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 18:39:25,199 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 18:39:25,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 18:39:25.426 UTC [12284] WARNING: there is no transaction in progress
2026-06-19 18:39:25,431 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 18:39:25,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 18:39:25,662 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 18:39:25,663 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:39:25,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 18:39:25,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 18:39:26,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rxv6.c.gcp-upstream.internal','N') ;"
2026-06-19 18:39:26,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:39:26,637 [I] Table x_portal_user does not exist in database ranger
2026-06-19 18:39:26,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:39:26,886 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 18:39:26,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:39:27,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:39:27,371 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 18:39:27,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 18:39:28.316 UTC [12392] WARNING: there is no transaction in progress
2026-06-19 18:39:28.332 UTC [12392] WARNING: there is no transaction in progress
2026-06-19 18:39:28.349 UTC [12392] WARNING: there is no transaction in progress
2026-06-19 18:39:28.364 UTC [12392] WARNING: there is no transaction in progress
2026-06-19 18:39:28.380 UTC [12392] WARNING: there is no transaction in progress
2026-06-19 18:39:28.397 UTC [12392] WARNING: there is no transaction in progress
2026-06-19 18:39:28.453 UTC [12392] WARNING: there is no transaction in progress
2026-06-19 18:39:28.459 UTC [12392] WARNING: there is no transaction in progress
2026-06-19 18:39:28.474 UTC [12392] WARNING: there is no transaction in progress
2026-06-19 18:39:28.484 UTC [12392] WARNING: there is no transaction in progress
2026-06-19 18:39:28.495 UTC [12392] WARNING: there is no transaction in progress
2026-06-19 18:39:28.508 UTC [12392] WARNING: there is no transaction in progress
2026-06-19 18:39:28.518 UTC [12392] WARNING: there is no transaction in progress
2026-06-19 18:39:28.525 UTC [12392] WARNING: there is no transaction in progress
2026-06-19 18:39:28.532 UTC [12392] WARNING: there is no transaction in progress
2026-06-19 18:39:28.539 UTC [12392] WARNING: there is no transaction in progress
2026-06-19 18:39:29,200 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 18:39:29,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 18:39:29,449 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 18:39:29,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 18:39:29,697 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 18:39:29,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 18:39:29,939 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 18:39:29,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 18:39:30,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 18:39:30,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rxv6.c.gcp-upstream.internal';"
2026-06-19 18:39:30,661 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 18:39:30,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 18:39:30,900 [I] DB_PATCHES have already been applied
I20260619 18:39:30.905457 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskqHyw_j/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 18:39:30.905527 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 18:39:30.908710 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:36003
Jun 19, 2026 6:39:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:39:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:39:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:39:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:39:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:39:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:39:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 18:39:31.894812 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.215s user 0.000s sys 0.009s
Jun 19, 2026 6:39:52 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:39:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 6:39:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:39:52 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:39:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:39:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:39:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:39:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:39:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:39:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:39:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:39:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:39:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:39:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:39:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:39:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:39:52 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:39:53 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 18:39:53.886107 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 18:39:53 dist-test-slave-rxv6 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894393, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 18:39:53.963372 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 18:39:53.972136 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 18:39:54.791157 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 18:39:54.791244 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-kms
I20260619 18:39:54.791275 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:58157
2026-06-19 18:39:54,943 [I] DB FLAVOR :POSTGRES
2026-06-19 18:39:54,943 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 18:39:54,943 [I] Checking connection
2026-06-19 18:39:54,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 18:39:55,191 [I] connection success
2026-06-19 18:39:55,192 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 18:39:55,192 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 18:39:55,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 18:39:55,449 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 18:39:55,449 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 18:39:55,449 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 18:39:55,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58157/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 18:39:55,696 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 18:39:56.126086 392 mini_ranger_kms.cc:310] Created kms service
I20260619 18:39:56.286752 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 18:39:56.329087 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 18:39:56 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894393, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 18:39:56.687542 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 18:39:56.687687 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskqHyw_j/build/debug/bin/postgresql.jar:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskqHyw_j/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 18:39:56.687718 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 18:39:56.691078 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12572
I20260619 18:39:56.691164 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:60987
Jun 19, 2026 6:39:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:39:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:39:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:39:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:39:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:39:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 18:39:58.729197 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.766s user 0.007s sys 0.128s
I20260619 18:39:58.729315 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 18:39:58.729379 392 mini_ranger_kms.cc:401] 127.0.98.20:60987/kms/v1/keys
Jun 19 18:39:58 dist-test-slave-rxv6 krb5kdc[12179](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894398, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 18:39:59 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894398, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 18:39:59 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894398, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 18:40:01 dist-test-slave-rxv6 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:40:01.877360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44097
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44097
--ranger_config_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60987/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:01.987313 12623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:01.987581 12623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:01.987619 12623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:01.991611 12623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:40:01.991685 12623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:01.991703 12623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:01.991717 12623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:40:01.991731 12623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:40:01.997479 12623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:60987/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44097
--ranger_config_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:01.998579 12623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:01.999647 12623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:02.009493 12628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:02.009498 12631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:02.009795 12623 server_base.cc:1034] running on GCE node
W20260619 18:40:02.009529 12629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:02.010516 12623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:02.011626 12623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:02.012859 12623 hybrid_clock.cc:648] HybridClock initialized: now 1781894402012842 us; error 34 us; skew 500 ppm
Jun 19 18:40:02 dist-test-slave-rxv6 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:02.021220 12623 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:40:02.022276 12623 webserver.cc:460] Webserver started at http://127.0.98.62:43455/ using document root <none> and password file <none>
I20260619 18:40:02.022846 12623 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:02.022907 12623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:02.023102 12623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:40:02 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 18:40:02.070150 12623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "1721f19404e74044960542637d56c74b"
format_stamp: "Formatted at 2026-06-19 18:40:02 on dist-test-slave-rxv6"
server_key: "d5c7f334fa674d8394ae90c6ea39b09d"
server_key_iv: "4e3bee8606ad2461ff3d1cdf78d53a12"
server_key_version: "kuduclusterkey@0"
I20260619 18:40:02.070752 12623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "1721f19404e74044960542637d56c74b"
format_stamp: "Formatted at 2026-06-19 18:40:02 on dist-test-slave-rxv6"
server_key: "d5c7f334fa674d8394ae90c6ea39b09d"
server_key_iv: "4e3bee8606ad2461ff3d1cdf78d53a12"
server_key_version: "kuduclusterkey@0"
I20260619 18:40:02.074493 12623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:40:02.088299 12638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:02.089648 12623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 18:40:02.089771 12623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "1721f19404e74044960542637d56c74b"
format_stamp: "Formatted at 2026-06-19 18:40:02 on dist-test-slave-rxv6"
server_key: "d5c7f334fa674d8394ae90c6ea39b09d"
server_key_iv: "4e3bee8606ad2461ff3d1cdf78d53a12"
server_key_version: "kuduclusterkey@0"
I20260619 18:40:02.089901 12623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:02.133371 12623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:02.136291 12623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:02.136482 12623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:02.143492 12623 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44097
I20260619 18:40:02.143509 12690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44097 every 8 connection(s)
I20260619 18:40:02.144435 12623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:40:02.144815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 12623
I20260619 18:40:02.144948 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:40:02.145246 392 external_mini_cluster.cc:1428] Setting key ffedd91ed04d67a9be84baecc0139ab7
I20260619 18:40:02.147461 12691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:02.152758 12691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b: Bootstrap starting.
Jun 19 18:40:02 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:02.155215 12691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:02.156098 12691 log.cc:824] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:02.158157 12691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b: No bootstrap required, opened a new log
I20260619 18:40:02.159690 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:02.146601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43484 (local address 127.0.98.62:44097)
0619 18:40:02.147028 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:02.147043 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:02.147079 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:40:02.147742 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:02.147759 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:02.148080 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:02.148403 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:02.149166 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:02.150267 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:02.151103 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:02.151560 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:02.151700 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:02.154440 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:02.154447 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:02.154462 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:02.154493 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:02.156760 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:02.157160 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:02.157165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:02.157171 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:02.157290 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:02.157599 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:02.157602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:02.157604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:02.158095 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:02.158280 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:02.158545 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 18:40:02.158805 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":130,"threads_started":1}
I20260619 18:40:02.161684 12691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1721f19404e74044960542637d56c74b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } }
I20260619 18:40:02.161962 12691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:02.162022 12691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1721f19404e74044960542637d56c74b, State: Initialized, Role: FOLLOWER
I20260619 18:40:02.162405 12691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1721f19404e74044960542637d56c74b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } }
I20260619 18:40:02.162524 12691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:40:02.162562 12691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:40:02.162624 12691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:02.163426 12691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1721f19404e74044960542637d56c74b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } }
I20260619 18:40:02.163712 12691 leader_election.cc:302] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1721f19404e74044960542637d56c74b; no voters:
I20260619 18:40:02.163966 12691 leader_election.cc:288] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:40:02.164072 12696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:02.164295 12696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [term 1 LEADER]: Becoming Leader. State: Replica: 1721f19404e74044960542637d56c74b, State: Running, Role: LEADER
I20260619 18:40:02.164646 12696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1721f19404e74044960542637d56c74b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } }
I20260619 18:40:02.165067 12691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:40:02.166206 12697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1721f19404e74044960542637d56c74b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1721f19404e74044960542637d56c74b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } } }
I20260619 18:40:02.166329 12697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:02.166649 12698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1721f19404e74044960542637d56c74b. Latest consensus state: current_term: 1 leader_uuid: "1721f19404e74044960542637d56c74b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1721f19404e74044960542637d56c74b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } } }
I20260619 18:40:02.166734 12705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:40:02.166771 12698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:02.167142 12691 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 18:40:02.171885 12705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:40:02.178604 12705 catalog_manager.cc:1347] Generated new cluster ID: addf86c11dce434d859b5402487392e3
I20260619 18:40:02.178666 12705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:40:02.202103 12705 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:40:02.203390 12705 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:40:02.211175 12705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b: Generated new TSK 0
I20260619 18:40:02.212045 12705 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:40:02.855134 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60987/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44097
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:02.980587 12736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:02.980854 12736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:02.980916 12736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:02.985515 12736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:02.985581 12736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:02.985662 12736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:40:02.992175 12736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:60987/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:02.993369 12736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:02.994666 12736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:40:03.004402 12736 server_base.cc:1034] running on GCE node
W20260619 18:40:03.004546 12746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:03.004703 12743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:03.004858 12744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:03.005241 12736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:03.005856 12736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:03.007050 12736 hybrid_clock.cc:648] HybridClock initialized: now 1781894403007006 us; error 33 us; skew 500 ppm
Jun 19 18:40:03 dist-test-slave-rxv6 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:03.009552 12736 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:40:03.010412 12736 webserver.cc:460] Webserver started at http://127.0.98.1:43267/ using document root <none> and password file <none>
I20260619 18:40:03.010979 12736 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:03.011044 12736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:03.011251 12736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:40:03 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 18:40:03.026402 12736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "20db83031a35401c84c52f130e11c3c9"
format_stamp: "Formatted at 2026-06-19 18:40:03 on dist-test-slave-rxv6"
server_key: "4aaa1894d04d971677d9a8333fd4b883"
server_key_iv: "96835bdc0f95b7bb28a1c948ac8e585e"
server_key_version: "kuduclusterkey@0"
I20260619 18:40:03.028158 12736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "20db83031a35401c84c52f130e11c3c9"
format_stamp: "Formatted at 2026-06-19 18:40:03 on dist-test-slave-rxv6"
server_key: "4aaa1894d04d971677d9a8333fd4b883"
server_key_iv: "96835bdc0f95b7bb28a1c948ac8e585e"
server_key_version: "kuduclusterkey@0"
I20260619 18:40:03.035317 12736 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260619 18:40:03.051864 12753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:03.053494 12736 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:40:03.053624 12736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "20db83031a35401c84c52f130e11c3c9"
format_stamp: "Formatted at 2026-06-19 18:40:03 on dist-test-slave-rxv6"
server_key: "4aaa1894d04d971677d9a8333fd4b883"
server_key_iv: "96835bdc0f95b7bb28a1c948ac8e585e"
server_key_version: "kuduclusterkey@0"
I20260619 18:40:03.053764 12736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:03.069519 12736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:03.073127 12736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:03.073374 12736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:03.074180 12736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:03.075248 12736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:03.075347 12736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:03.075460 12736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:03.075536 12736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:03.089149 12736 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41929
I20260619 18:40:03.089182 12868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41929 every 8 connection(s)
I20260619 18:40:03.090230 12736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:40:03.095474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 12736
I20260619 18:40:03.095620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:40:03.095906 392 external_mini_cluster.cc:1428] Setting key 608032befa67bd3c5df3821915fe92a9
Jun 19 18:40:03 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:03.110082 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:03.092618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44097 (local address 127.0.98.1:44751)
0619 18:40:03.094211 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:03.094269 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:03.095305 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:03.096186 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:03.096195 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:03.097557 (+ 1362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:03.098261 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:03.098291 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:03.099533 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:03.099536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:03.100989 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:03.101000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:03.101738 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:03.101742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:03.101973 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:03.102672 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:03.102694 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:03.104431 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:03.106646 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:03.106652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:03.106669 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:03.107017 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:03.107327 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:03.107330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:03.107332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:03.107504 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:03.108105 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:03.108112 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:03.108441 (+ 329us) client_negotiation.cc:769] Sending connection context
0619 18:40:03.108636 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 18:40:03.109005 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1359,"thread_start_us":121,"threads_started":1}
I20260619 18:40:03.110852 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:03.092189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44751 (local address 127.0.98.62:44097)
0619 18:40:03.095740 (+ 3551us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:03.095748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:03.095762 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:40:03.095805 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:03.095809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:03.095858 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:03.098424 (+ 2566us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:03.098532 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:03.099388 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:03.101122 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:03.101473 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:03.101564 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:03.104631 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:03.104634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:03.104637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:03.104664 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:03.106516 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:03.107131 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:03.107135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:03.107137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:03.107229 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:03.107626 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:03.107629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:03.107631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:03.107850 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:03.108047 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:03.110552 (+ 2505us) server_negotiation.cc:299] Negotiation successful
0619 18:40:03.110682 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3470,"thread_start_us":115,"threads_started":1}
I20260619 18:40:03.114341 12869 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44097
I20260619 18:40:03.114595 12869 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:03.115198 12869 heartbeater.cc:511] Master 127.0.98.62:44097 requested a full tablet report, sending...
I20260619 18:40:03.116998 12655 ts_manager.cc:194] Registered new tserver with Master: 20db83031a35401c84c52f130e11c3c9 (127.0.98.1:41929)
I20260619 18:40:03.118521 12655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44751
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:40:03.140617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60987/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44097
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:40:03 dist-test-slave-rxv6 krb5kdc[12179](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 18:40:03.273463 12875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:03.273748 12875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:03.273806 12875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:03.278260 12875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:03.278333 12875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:03.278417 12875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:40:03.283859 12875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:60987/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:03.285034 12875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:03.286099 12875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:03.296589 12882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:03.296589 12884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:03.297124 12881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:03.297370 12875 server_base.cc:1034] running on GCE node
I20260619 18:40:03.297806 12875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:03.298496 12875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:03.299662 12875 hybrid_clock.cc:648] HybridClock initialized: now 1781894403299633 us; error 51 us; skew 500 ppm
Jun 19 18:40:03 dist-test-slave-rxv6 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:03.302397 12875 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:40:03.303318 12875 webserver.cc:460] Webserver started at http://127.0.98.2:36699/ using document root <none> and password file <none>
I20260619 18:40:03.303884 12875 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:03.303948 12875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:03.304179 12875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:40:03 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 18:40:03.317941 12875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "1a516def1efa4b5cb3f92aa1e5935f7e"
format_stamp: "Formatted at 2026-06-19 18:40:03 on dist-test-slave-rxv6"
server_key: "93556c21bfddbcd6bd5e607ab85e3fd6"
server_key_iv: "8a7ca9d8a0850cd188e3969b6f6c277e"
server_key_version: "kuduclusterkey@0"
I20260619 18:40:03.318861 12875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1a516def1efa4b5cb3f92aa1e5935f7e"
format_stamp: "Formatted at 2026-06-19 18:40:03 on dist-test-slave-rxv6"
server_key: "93556c21bfddbcd6bd5e607ab85e3fd6"
server_key_iv: "8a7ca9d8a0850cd188e3969b6f6c277e"
server_key_version: "kuduclusterkey@0"
I20260619 18:40:03.332319 12875 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.000s
I20260619 18:40:03.352644 12891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:03.359764 12875 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.000s
I20260619 18:40:03.359964 12875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "1a516def1efa4b5cb3f92aa1e5935f7e"
format_stamp: "Formatted at 2026-06-19 18:40:03 on dist-test-slave-rxv6"
server_key: "93556c21bfddbcd6bd5e607ab85e3fd6"
server_key_iv: "8a7ca9d8a0850cd188e3969b6f6c277e"
server_key_version: "kuduclusterkey@0"
I20260619 18:40:03.360127 12875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:03.393868 12875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:03.397967 12875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:03.398227 12875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:03.398912 12875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:03.400000 12875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:03.400108 12875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:03.400203 12875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:03.400266 12875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:03.412999 12875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39171
I20260619 18:40:03.414270 12875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:40:03.416805 13004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39171 every 8 connection(s)
I20260619 18:40:03.422086 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 12875
I20260619 18:40:03.422219 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:40:03.422487 392 external_mini_cluster.cc:1428] Setting key b97f460b95f796fc97744a50927415fc
Jun 19 18:40:03 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:03.437844 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:03.420995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41553 (local address 127.0.98.62:44097)
0619 18:40:03.421143 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:03.421148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:03.423667 (+ 2519us) server_negotiation.cc:407] Connection header received
0619 18:40:03.424945 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:03.424948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:03.425003 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:03.425061 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:03.426798 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:03.427583 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:03.429375 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:03.429761 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:03.429847 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:03.433224 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:03.433229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:03.433233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:03.433268 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:03.435123 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:03.435841 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:03.435846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:03.435848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:03.435945 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:03.436396 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:03.436399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:03.436401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:03.436659 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:03.436748 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:03.437523 (+ 775us) server_negotiation.cc:299] Negotiation successful
0619 18:40:03.437670 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:40:03.439888 13007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:03.421302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44097 (local address 127.0.98.2:41553)
0619 18:40:03.423495 (+ 2193us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:03.423559 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:03.424696 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:03.425496 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:03.425507 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:03.425876 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:03.426617 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:03.426637 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:03.427749 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:03.427752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:03.429217 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:03.429230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:03.430101 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:03.430105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:03.430328 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:03.431018 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:03.431041 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:03.432977 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:03.435286 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:03.435293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:03.435314 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:03.435712 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:03.436070 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:03.436073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:03.436075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:03.436279 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:03.436821 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:03.436828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:03.437194 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 18:40:03.438367 (+ 1173us) client_negotiation.cc:240] Negotiation successful
0619 18:40:03.438759 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1943,"thread_start_us":159,"threads_started":1}
I20260619 18:40:03.441550 13005 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44097
I20260619 18:40:03.441807 13005 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:03.442437 13005 heartbeater.cc:511] Master 127.0.98.62:44097 requested a full tablet report, sending...
I20260619 18:40:03.443668 12655 ts_manager.cc:194] Registered new tserver with Master: 1a516def1efa4b5cb3f92aa1e5935f7e (127.0.98.2:39171)
I20260619 18:40:03.444438 12655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41553
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:40:03.471320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60987/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44097
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:03.630594 13010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:03.631022 13010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:03.631132 13010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:03.637353 13010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:03.637508 13010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:03.637652 13010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:40:03.646009 13010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:60987/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:03.647588 13010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:03.649119 13010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:03.661623 13016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:03.662220 13015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:03.661623 13018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:03.663450 13010 server_base.cc:1034] running on GCE node
I20260619 18:40:03.663952 13010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:03.664815 13010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:03.666257 13010 hybrid_clock.cc:648] HybridClock initialized: now 1781894403666190 us; error 81 us; skew 500 ppm
Jun 19 18:40:03 dist-test-slave-rxv6 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:03.670156 13010 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:40:03.671470 13010 webserver.cc:460] Webserver started at http://127.0.98.3:37981/ using document root <none> and password file <none>
I20260619 18:40:03.672230 13010 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:03.672366 13010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:03.672632 13010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 18:40:03 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 18:40:03.696141 13010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "db631fac145841c7914b447fc8d98bd9"
format_stamp: "Formatted at 2026-06-19 18:40:03 on dist-test-slave-rxv6"
server_key: "0426580dc9868754bcb48c097eb3bb00"
server_key_iv: "0d3d9f75420c2eaab13dacd01dd11240"
server_key_version: "kuduclusterkey@0"
I20260619 18:40:03.698186 13010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "db631fac145841c7914b447fc8d98bd9"
format_stamp: "Formatted at 2026-06-19 18:40:03 on dist-test-slave-rxv6"
server_key: "0426580dc9868754bcb48c097eb3bb00"
server_key_iv: "0d3d9f75420c2eaab13dacd01dd11240"
server_key_version: "kuduclusterkey@0"
I20260619 18:40:03.704157 13010 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260619 18:40:03.733093 13025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:03.735014 13010 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:40:03.735255 13010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "db631fac145841c7914b447fc8d98bd9"
format_stamp: "Formatted at 2026-06-19 18:40:03 on dist-test-slave-rxv6"
server_key: "0426580dc9868754bcb48c097eb3bb00"
server_key_iv: "0d3d9f75420c2eaab13dacd01dd11240"
server_key_version: "kuduclusterkey@0"
I20260619 18:40:03.735482 13010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Jun 19 18:40:03 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 18:40:03.768785 13010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:03.772584 13010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:03.772884 13010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:03.773646 13010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:03.774789 13010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:03.774887 13010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:03.774976 13010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:03.775039 13010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:03.784758 13010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41617
I20260619 18:40:03.789829 13010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:40:03.788635 13138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41617 every 8 connection(s)
I20260619 18:40:03.795328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 13010
I20260619 18:40:03.795579 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:40:03.796146 392 external_mini_cluster.cc:1428] Setting key 2e0c7227e3acad7e969ea6235499912a
Jun 19 18:40:03 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:03.816560 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:03.792834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34513 (local address 127.0.98.62:44097)
0619 18:40:03.792971 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:03.792976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:03.797995 (+ 5019us) server_negotiation.cc:407] Connection header received
0619 18:40:03.802436 (+ 4441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:03.802441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:03.802550 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:03.802644 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:03.804633 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:03.805471 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:03.807474 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:03.807830 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:03.807910 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:03.811178 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:03.811182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:03.811185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:03.811216 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:03.812971 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:03.814081 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:03.814111 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:03.814113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:03.814246 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:03.815178 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:03.815183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:03.815185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:03.815414 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:03.815489 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:03.816256 (+ 767us) server_negotiation.cc:299] Negotiation successful
0619 18:40:03.816388 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:40:03.818462 13141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:03.793725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44097 (local address 127.0.98.3:34513)
0619 18:40:03.797537 (+ 3812us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:03.797728 (+ 191us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:03.801674 (+ 3946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:03.803278 (+ 1604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:03.803289 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:03.803676 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:03.804455 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:03.804475 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:03.805635 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:03.805639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:03.807332 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:03.807345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:03.808126 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:03.808130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:03.808336 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:03.809023 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:03.809045 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:03.810938 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:03.813214 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:03.813222 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:03.813242 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:03.813900 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:03.814754 (+ 854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:03.814758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:03.814759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:03.815066 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:03.815566 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:03.815573 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:03.815978 (+ 405us) client_negotiation.cc:769] Sending connection context
0619 18:40:03.817006 (+ 1028us) client_negotiation.cc:240] Negotiation successful
0619 18:40:03.817370 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3084,"thread_start_us":201,"threads_started":1}
I20260619 18:40:03.820101 13139 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44097
I20260619 18:40:03.820351 13139 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:03.820933 13139 heartbeater.cc:511] Master 127.0.98.62:44097 requested a full tablet report, sending...
I20260619 18:40:03.822131 12655 ts_manager.cc:194] Registered new tserver with Master: db631fac145841c7914b447fc8d98bd9 (127.0.98.3:41617)
I20260619 18:40:03.822872 12655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34513
I20260619 18:40:03.828307 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:40:03.829149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 12736
I20260619 18:40:03.838302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 12875
I20260619 18:40:03.844612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 13010
I20260619 18:40:03.853495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 12623
I20260619 18:40:03.860628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44097
--webserver_interface=127.0.98.62
--webserver_port=43455
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44097
--ranger_config_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60987/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60987/kms with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:03.997326 13142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:03.997598 13142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:03.997637 13142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:04.001657 13142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:40:04.001732 13142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:04.001751 13142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:04.001766 13142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:40:04.001780 13142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:40:04.007278 13142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60987/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44097
--ranger_config_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:04.008268 13142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:04.009232 13142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:04.017349 13148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:04.017388 13142 server_base.cc:1034] running on GCE node
W20260619 18:40:04.017325 13147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:04.017338 13150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:04.018071 13142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:04.019176 13142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:04.020339 13142 hybrid_clock.cc:648] HybridClock initialized: now 1781894404020312 us; error 41 us; skew 500 ppm
Jun 19 18:40:04 dist-test-slave-rxv6 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:04.023175 13142 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:40:04.024065 13142 webserver.cc:460] Webserver started at http://127.0.98.62:43455/ using document root <none> and password file <none>
I20260619 18:40:04.024596 13142 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:04.024663 13142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:40:04 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 18:40:04.080654 13142 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 18:40:04.083449 13158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:04.084769 13142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:40:04.084896 13142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "1721f19404e74044960542637d56c74b"
format_stamp: "Formatted at 2026-06-19 18:40:02 on dist-test-slave-rxv6"
server_key: "d5c7f334fa674d8394ae90c6ea39b09d"
server_key_iv: "4e3bee8606ad2461ff3d1cdf78d53a12"
server_key_version: "kuduclusterkey@0"
I20260619 18:40:04.085436 13142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:04.097632 13142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:04.100567 13142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:04.100755 13142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:04.108146 13142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44097
I20260619 18:40:04.108201 13210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44097 every 8 connection(s)
I20260619 18:40:04.109234 13142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:40:04.113533 13211 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:40:04.115228 13211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b: Bootstrap starting.
I20260619 18:40:04.118745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 13142
I20260619 18:40:04.119223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60987/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41929
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44097
--webserver_port=43267
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60987/kms with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:40:04.131853 13211 log.cc:824] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:04.138676 13211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:40:04.139055 13211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b: Bootstrap complete.
I20260619 18:40:04.143190 13211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1721f19404e74044960542637d56c74b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } }
I20260619 18:40:04.143581 13211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1721f19404e74044960542637d56c74b, State: Initialized, Role: FOLLOWER
I20260619 18:40:04.144080 13211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1721f19404e74044960542637d56c74b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } }
I20260619 18:40:04.144205 13211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:40:04.144268 13211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:40:04.144356 13211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:40:04.146282 13211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1721f19404e74044960542637d56c74b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } }
I20260619 18:40:04.146592 13211 leader_election.cc:302] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1721f19404e74044960542637d56c74b; no voters:
I20260619 18:40:04.146867 13211 leader_election.cc:288] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:40:04.147011 13215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:40:04.147258 13215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [term 2 LEADER]: Becoming Leader. State: Replica: 1721f19404e74044960542637d56c74b, State: Running, Role: LEADER
I20260619 18:40:04.147658 13215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1721f19404e74044960542637d56c74b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } }
I20260619 18:40:04.148023 13211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:40:04.149047 13217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1721f19404e74044960542637d56c74b. Latest consensus state: current_term: 2 leader_uuid: "1721f19404e74044960542637d56c74b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1721f19404e74044960542637d56c74b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } } }
I20260619 18:40:04.149163 13217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:04.149817 13216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1721f19404e74044960542637d56c74b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1721f19404e74044960542637d56c74b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } } }
I20260619 18:40:04.149899 13216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:04.150187 13211 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 18:40:04.150247 13224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:40:04.156316 13224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:40:04.157146 13224 catalog_manager.cc:1259] Loaded cluster ID: addf86c11dce434d859b5402487392e3
I20260619 18:40:04.157279 13224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:40:04.159910 13224 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:40:04.160670 13224 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b: Loaded TSK: 0
I20260619 18:40:04.161983 13224 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:40:04.256261 13213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:04.256712 13213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:04.256824 13213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:04.263067 13213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:04.263175 13213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:04.263268 13213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:40:04.271468 13213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60987/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43267
--tserver_master_addrs=127.0.98.62:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:04.272874 13213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:04.274295 13213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:04.287125 13252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:04.287887 13249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:04.288465 13213 server_base.cc:1034] running on GCE node
W20260619 18:40:04.289099 13250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:04.289642 13213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:04.290503 13213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:04.291673 13213 hybrid_clock.cc:648] HybridClock initialized: now 1781894404291641 us; error 50 us; skew 500 ppm
Jun 19 18:40:04 dist-test-slave-rxv6 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:04.295470 13213 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:40:04.296603 13213 webserver.cc:460] Webserver started at http://127.0.98.1:43267/ using document root <none> and password file <none>
I20260619 18:40:04.297360 13213 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:04.297485 13213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:40:04 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 18:40:04.326067 13213 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:40:04.329149 13260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:04.330729 13213 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:40:04.330860 13213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "20db83031a35401c84c52f130e11c3c9"
format_stamp: "Formatted at 2026-06-19 18:40:03 on dist-test-slave-rxv6"
server_key: "4aaa1894d04d971677d9a8333fd4b883"
server_key_iv: "96835bdc0f95b7bb28a1c948ac8e585e"
server_key_version: "kuduclusterkey@0"
I20260619 18:40:04.331599 13213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:04.371136 13213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:04.375399 13213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:04.375669 13213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:04.376466 13213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:04.378014 13213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:04.378152 13213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:04.378337 13213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:04.378376 13213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:04.389485 13213 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41929
I20260619 18:40:04.389746 13373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41929 every 8 connection(s)
I20260619 18:40:04.390954 13213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:40:04.398787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 13213
I20260619 18:40:04.399271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60987/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39171
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44097
--webserver_port=36699
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60987/kms with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 18:40:04 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:04.418363 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:04.393334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44423 (local address 127.0.98.62:44097)
0619 18:40:04.393892 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:04.395577 (+ 1685us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:04.395626 (+ 49us) server_negotiation.cc:407] Connection header received
0619 18:40:04.395907 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:04.395920 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:04.396337 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:04.396678 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:04.398039 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:04.399281 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:04.408161 (+ 8880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:04.409096 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:04.409346 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:04.412405 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:04.412413 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:04.412434 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:04.412470 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:04.414552 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:04.415176 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:04.415182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:04.415188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:04.415324 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:04.415720 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:04.415723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:04.415725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:04.416181 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:04.416341 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:04.417085 (+ 744us) server_negotiation.cc:299] Negotiation successful
0619 18:40:04.417370 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":109,"threads_started":1}
I20260619 18:40:04.418376 13377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:04.393486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44097 (local address 127.0.98.1:44423)
0619 18:40:04.394196 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:04.394250 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:04.395272 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:04.397143 (+ 1871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:04.397150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:04.397417 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:04.397863 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:04.397876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:04.399480 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:04.399484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:04.400761 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:04.400774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:04.409365 (+ 8591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:04.409372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:04.409597 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:04.410385 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:04.410408 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:04.412145 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:04.414683 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:04.414689 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:04.414705 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:04.415033 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:04.415431 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:04.415434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:04.415436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:04.415603 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:04.416343 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:04.416351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:04.416706 (+ 355us) client_negotiation.cc:769] Sending connection context
0619 18:40:04.416931 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 18:40:04.417291 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":116,"threads_started":1}
I20260619 18:40:04.421511 13374 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44097
I20260619 18:40:04.421782 13374 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:04.422374 13374 heartbeater.cc:511] Master 127.0.98.62:44097 requested a full tablet report, sending...
W20260619 18:40:04.424804 13374 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44097 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:40:04.424932 13374 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:04.425107 13374 heartbeater.cc:511] Master 127.0.98.62:44097 requested a full tablet report, sending...
I20260619 18:40:04.426023 13374 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:04.426193 13374 heartbeater.cc:511] Master 127.0.98.62:44097 requested a full tablet report, sending...
W20260619 18:40:04.426994 13374 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:40:04.528366 13378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:04.528654 13378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:04.528702 13378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:04.532898 13378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:04.532972 13378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:04.533032 13378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:40:04.538690 13378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60987/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36699
--tserver_master_addrs=127.0.98.62:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:04.539764 13378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:04.540787 13378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:04.555055 13383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:04.555055 13384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:04.555794 13386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:04.556041 13378 server_base.cc:1034] running on GCE node
I20260619 18:40:04.556530 13378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:04.557385 13378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:04.558606 13378 hybrid_clock.cc:648] HybridClock initialized: now 1781894404558581 us; error 34 us; skew 500 ppm
Jun 19 18:40:04 dist-test-slave-rxv6 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:04.562335 13378 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:40:04.563405 13378 webserver.cc:460] Webserver started at http://127.0.98.2:36699/ using document root <none> and password file <none>
I20260619 18:40:04.564075 13378 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:04.564167 13378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:40:04 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 18:40:04.596192 13378 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:40:04.599437 13394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:04.601233 13378 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260619 18:40:04.601351 13378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "1a516def1efa4b5cb3f92aa1e5935f7e"
format_stamp: "Formatted at 2026-06-19 18:40:03 on dist-test-slave-rxv6"
server_key: "93556c21bfddbcd6bd5e607ab85e3fd6"
server_key_iv: "8a7ca9d8a0850cd188e3969b6f6c277e"
server_key_version: "kuduclusterkey@0"
I20260619 18:40:04.601991 13378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:04.627209 13378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:04.630903 13378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:04.631168 13378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:04.631893 13378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:04.633136 13378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:04.633236 13378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 18:40:04.633322 13378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:04.633370 13378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:04.643507 13378 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39171
I20260619 18:40:04.644876 13378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:40:04.650347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 13378
I20260619 18:40:04.650746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60987/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41617
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44097
--webserver_port=37981
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60987/kms with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:40:04.652621 13508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39171 every 8 connection(s)
Jun 19 18:40:04 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:04.674887 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:04.656438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60471 (local address 127.0.98.62:44097)
0619 18:40:04.656563 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:04.656566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:04.658894 (+ 2328us) server_negotiation.cc:407] Connection header received
0619 18:40:04.660328 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:04.660332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:04.660400 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:04.660505 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:04.662188 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:04.663049 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:04.664892 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:04.665280 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:04.665384 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:04.670403 (+ 5019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:04.670408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:04.670411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:04.670444 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:04.672209 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:04.672965 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:04.672969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:04.672971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:04.673079 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:04.673506 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:04.673509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:04.673511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:04.673776 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:04.673888 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:04.674564 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 18:40:04.674704 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 18:40:04.675940 13512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:04.655251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44097 (local address 127.0.98.2:60471)
0619 18:40:04.658662 (+ 3411us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:04.658751 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:04.660008 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:04.660744 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:04.660755 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:04.661126 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:04.661959 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:04.661980 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:04.663283 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:04.663287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:04.664720 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:04.664731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:04.665568 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:04.665572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:04.665782 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:04.666546 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:04.666567 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:04.670172 (+ 3605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:04.672368 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:04.672375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:04.672393 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:04.672824 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:04.673182 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:04.673185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:04.673187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:04.673384 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:04.673884 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:04.673891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:04.674269 (+ 378us) client_negotiation.cc:769] Sending connection context
0619 18:40:04.674466 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 18:40:04.674807 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1753,"thread_start_us":159,"threads_started":1}
I20260619 18:40:04.677315 13509 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44097
I20260619 18:40:04.677567 13509 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:04.678164 13509 heartbeater.cc:511] Master 127.0.98.62:44097 requested a full tablet report, sending...
W20260619 18:40:04.679888 13509 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44097 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 18:40:04.680023 13509 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:04.680255 13509 heartbeater.cc:511] Master 127.0.98.62:44097 requested a full tablet report, sending...
I20260619 18:40:04.681206 13509 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:04.681394 13509 heartbeater.cc:511] Master 127.0.98.62:44097 requested a full tablet report, sending...
W20260619 18:40:04.682542 13509 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:40:04.784919 13511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:04.785255 13511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:04.785321 13511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:04.789685 13511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:04.789813 13511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:04.789942 13511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:40:04.797176 13511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:60987/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37981
--tserver_master_addrs=127.0.98.62:44097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:04.798384 13511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:04.799372 13511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:04.810369 13518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:04.810369 13520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:04.810799 13517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:04.810889 13511 server_base.cc:1034] running on GCE node
I20260619 18:40:04.811303 13511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:04.811933 13511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:04.813139 13511 hybrid_clock.cc:648] HybridClock initialized: now 1781894404813125 us; error 40 us; skew 500 ppm
Jun 19 18:40:04 dist-test-slave-rxv6 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:04.816313 13511 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:40:04.817438 13511 webserver.cc:460] Webserver started at http://127.0.98.3:37981/ using document root <none> and password file <none>
I20260619 18:40:04.818164 13511 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:04.818281 13511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 18:40:04 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 18:40:04.846851 13511 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.002s sys 0.000s
I20260619 18:40:04.851091 13528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:04.852886 13511 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260619 18:40:04.853040 13511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "db631fac145841c7914b447fc8d98bd9"
format_stamp: "Formatted at 2026-06-19 18:40:03 on dist-test-slave-rxv6"
server_key: "0426580dc9868754bcb48c097eb3bb00"
server_key_iv: "0d3d9f75420c2eaab13dacd01dd11240"
server_key_version: "kuduclusterkey@0"
I20260619 18:40:04.853883 13511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:04.887992 13511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:04.891549 13511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:04.891757 13511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:04.892489 13511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:04.893574 13511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:04.893637 13511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:04.893690 13511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:04.893719 13511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:04.904839 13511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41617
I20260619 18:40:04.906296 13511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:40:04.906508 13641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41617 every 8 connection(s)
I20260619 18:40:04.908735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 13511
Jun 19 18:40:04 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:04.922044 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:04.910485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56669 (local address 127.0.98.62:44097)
0619 18:40:04.911024 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:04.911030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:04.911053 (+ 23us) server_negotiation.cc:407] Connection header received
0619 18:40:04.911101 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:04.911105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:04.911157 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:04.911265 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:04.912368 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:04.913128 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:04.914614 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:04.915053 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:04.915165 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:04.918201 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:04.918205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:04.918207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:04.918228 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:04.919813 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:04.920457 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:04.920461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:04.920463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:04.920538 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:04.920886 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:04.920889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:04.920890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:04.921096 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:04.921175 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:04.921709 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 18:40:04.921834 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":447,"spinlock_wait_cycles":2304}
I20260619 18:40:04.922888 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:04.909254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44097 (local address 127.0.98.3:56669)
0619 18:40:04.909862 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:04.909944 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:04.910892 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:04.911451 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:04.911460 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:04.911754 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:04.912220 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:04.912233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:04.913266 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:04.913279 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:04.914499 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:04.914508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:04.915213 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:04.915217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:04.915390 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:04.916522 (+ 1132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:04.916542 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:04.918053 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:04.919928 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:04.919935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:04.919953 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:04.920343 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:04.920624 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:04.920627 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:04.920629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:04.920781 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:04.921189 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:04.921196 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:04.921472 (+ 276us) client_negotiation.cc:769] Sending connection context
0619 18:40:04.921636 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 18:40:04.921970 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":181,"threads_started":1}
I20260619 18:40:04.924017 13642 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44097
I20260619 18:40:04.924247 13642 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:04.924721 13642 heartbeater.cc:511] Master 127.0.98.62:44097 requested a full tablet report, sending...
I20260619 18:40:04.923430 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:04.911492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.0.98.62:44097)
0619 18:40:04.912450 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:04.912454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:04.912463 (+ 9us) server_negotiation.cc:407] Connection header received
0619 18:40:04.912494 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:04.912496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:04.912529 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:04.912601 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:04.914618 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:04.915208 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:04.916420 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:04.916734 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:04.916837 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:04.918692 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:04.918696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:04.918699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:04.918744 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:04.920394 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:04.920825 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:04.920828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:04.920830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:04.920923 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:04.922010 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:04.922014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:04.922016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:04.922236 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:04.923028 (+ 792us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:04.923138 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 18:40:04.923262 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":901,"spinlock_wait_cycles":2944,"thread_start_us":85,"threads_started":1}
I20260619 18:40:04.926070 13172 ts_manager.cc:194] Registered new tserver with Master: db631fac145841c7914b447fc8d98bd9 (127.0.98.3:41617)
I20260619 18:40:04.927618 13172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56669
Jun 19 18:40:05 dist-test-slave-rxv6 krb5kdc[12179](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781894405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:05.427212 13374 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:05.427543 13374 heartbeater.cc:511] Master 127.0.98.62:44097 requested a full tablet report, sending...
I20260619 18:40:05.428390 13172 ts_manager.cc:194] Registered new tserver with Master: 20db83031a35401c84c52f130e11c3c9 (127.0.98.1:41929)
I20260619 18:40:05.429258 13172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44423
Jun 19 18:40:05 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 18:40:05.682837 13509 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:05.683130 13509 heartbeater.cc:511] Master 127.0.98.62:44097 requested a full tablet report, sending...
I20260619 18:40:05.683926 13172 ts_manager.cc:194] Registered new tserver with Master: 1a516def1efa4b5cb3f92aa1e5935f7e (127.0.98.2:39171)
I20260619 18:40:05.684685 13172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60471
I20260619 18:40:05.690897 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:40:05.702474 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:05.692491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.0.98.62:44097)
0619 18:40:05.693119 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:05.693124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:05.693425 (+ 301us) server_negotiation.cc:407] Connection header received
0619 18:40:05.693465 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:05.693468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:05.693514 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:05.693590 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:05.694491 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.695320 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.696497 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.696859 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.696945 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:05.698611 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:05.698616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:05.698618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:05.698643 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:05.700342 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:05.700808 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:05.700811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:05.700813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:05.700898 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:05.701165 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:05.701169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:05.701170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:05.701379 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:05.701499 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:05.702164 (+ 665us) server_negotiation.cc:299] Negotiation successful
0619 18:40:05.702291 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":68,"threads_started":1}
I20260619 18:40:05.706874 13172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:40:05.711660 13172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:40:05.732174 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:05.723930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41929 (local address 127.0.0.1:33444)
0619 18:40:05.726051 (+ 2121us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:05.726064 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:05.726147 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:05.726600 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:05.726603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:05.726620 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:05.726715 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:05.726721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.728055 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.728058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:05.730321 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:05.730332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.731625 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.731628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:05.731746 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:05.731794 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 18:40:05.731894 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 18:40:05.732012 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2059,"thread_start_us":89,"threads_started":1}
I20260619 18:40:05.732319 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:05.724427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.0.98.2:39171)
0619 18:40:05.725339 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:05.725346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:05.725368 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:40:05.725439 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:05.725444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:05.725648 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:05.725795 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:05.728111 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.729220 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.730721 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.731499 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.731580 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:05.731712 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:05.732065 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 18:40:05.732173 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":109,"threads_started":1}
I20260619 18:40:05.732831 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:05.723581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39171 (local address 127.0.0.1:45068)
0619 18:40:05.724912 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:05.724961 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:05.725084 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:05.725819 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:05.725825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:05.725851 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:05.726040 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:05.726048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.729393 (+ 3345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.729396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:05.730571 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:05.730583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.731808 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.731811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:05.731913 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:05.731934 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:40:05.732614 (+ 680us) client_negotiation.cc:240] Negotiation successful
0619 18:40:05.732698 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"thread_start_us":101,"threads_started":1}
I20260619 18:40:05.734097 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:05.724100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33444 (local address 127.0.98.1:41929)
0619 18:40:05.725081 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:05.725089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:05.726159 (+ 1070us) server_negotiation.cc:407] Connection header received
0619 18:40:05.726235 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:05.726240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:05.726484 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:05.726630 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:05.726793 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.727937 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.730457 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.731325 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.731425 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:05.731565 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:05.733818 (+ 2253us) server_negotiation.cc:299] Negotiation successful
0619 18:40:05.733959 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":87,"threads_started":1}
I20260619 18:40:05.736208 13443 tablet_service.cc:1467] Processing CreateTablet for tablet 350a470ad70e484b9358ad5f7b9edbb8 (DEFAULT_TABLE table=test-table [id=c76bd0ad5dd34e67901c101afe5fee44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:05.737357 13443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350a470ad70e484b9358ad5f7b9edbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:05.737658 13308 tablet_service.cc:1467] Processing CreateTablet for tablet 350a470ad70e484b9358ad5f7b9edbb8 (DEFAULT_TABLE table=test-table [id=c76bd0ad5dd34e67901c101afe5fee44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:05.738683 13308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350a470ad70e484b9358ad5f7b9edbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:05.739351 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:05.724315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41617 (local address 127.0.0.1:36092)
0619 18:40:05.732827 (+ 8512us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:05.732840 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:05.733734 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:05.734415 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:05.734418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:05.734434 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:05.734504 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:05.734509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.735830 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.735833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:05.736909 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:05.736919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.738983 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.738986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:05.739068 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:05.739090 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:40:05.739156 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 18:40:05.739226 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8447,"thread_start_us":80,"threads_started":1}
I20260619 18:40:05.739737 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:05.724691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.0.98.3:41617)
0619 18:40:05.726363 (+ 1672us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:05.726369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:05.733650 (+ 7281us) server_negotiation.cc:407] Connection header received
0619 18:40:05.733899 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:05.733924 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:05.734133 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:05.734271 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:05.734631 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.735696 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.737041 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.738679 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.738776 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:05.738921 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:05.739516 (+ 595us) server_negotiation.cc:299] Negotiation successful
0619 18:40:05.739608 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1573,"thread_start_us":88,"threads_started":1}
I20260619 18:40:05.742514 13576 tablet_service.cc:1467] Processing CreateTablet for tablet 350a470ad70e484b9358ad5f7b9edbb8 (DEFAULT_TABLE table=test-table [id=c76bd0ad5dd34e67901c101afe5fee44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:05.743508 13576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 350a470ad70e484b9358ad5f7b9edbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:05.746690 13675 tablet_bootstrap.cc:492] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e: Bootstrap starting.
I20260619 18:40:05.748970 13675 tablet_bootstrap.cc:654] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:05.749791 13675 log.cc:824] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:05.750535 13676 tablet_bootstrap.cc:492] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9: Bootstrap starting.
I20260619 18:40:05.752859 13676 tablet_bootstrap.cc:654] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:05.753525 13678 tablet_bootstrap.cc:492] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9: Bootstrap starting.
I20260619 18:40:05.753734 13676 log.cc:824] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:05.756134 13675 tablet_bootstrap.cc:492] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e: No bootstrap required, opened a new log
I20260619 18:40:05.756392 13675 ts_tablet_manager.cc:1397] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 18:40:05.756759 13678 tablet_bootstrap.cc:654] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:05.757632 13678 log.cc:824] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:05.760345 13676 tablet_bootstrap.cc:492] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9: No bootstrap required, opened a new log
I20260619 18:40:05.760603 13676 ts_tablet_manager.cc:1397] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 18:40:05.760841 13675 raft_consensus.cc:348] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a516def1efa4b5cb3f92aa1e5935f7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39171 } } peers { permanent_uuid: "db631fac145841c7914b447fc8d98bd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41617 } } peers { permanent_uuid: "20db83031a35401c84c52f130e11c3c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41929 } }
I20260619 18:40:05.761140 13675 raft_consensus.cc:374] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:05.761188 13675 raft_consensus.cc:729] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a516def1efa4b5cb3f92aa1e5935f7e, State: Initialized, Role: FOLLOWER
I20260619 18:40:05.761639 13675 consensus_queue.cc:260] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a516def1efa4b5cb3f92aa1e5935f7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39171 } } peers { permanent_uuid: "db631fac145841c7914b447fc8d98bd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41617 } } peers { permanent_uuid: "20db83031a35401c84c52f130e11c3c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41929 } }
I20260619 18:40:05.762728 13675 ts_tablet_manager.cc:1428] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:40:05.763108 13509 heartbeater.cc:503] Master 127.0.98.62:44097 was elected leader, sending a full tablet report...
I20260619 18:40:05.767071 13676 raft_consensus.cc:348] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a516def1efa4b5cb3f92aa1e5935f7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39171 } } peers { permanent_uuid: "db631fac145841c7914b447fc8d98bd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41617 } } peers { permanent_uuid: "20db83031a35401c84c52f130e11c3c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41929 } }
I20260619 18:40:05.767417 13676 raft_consensus.cc:374] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:05.767493 13676 raft_consensus.cc:729] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20db83031a35401c84c52f130e11c3c9, State: Initialized, Role: FOLLOWER
I20260619 18:40:05.767997 13676 consensus_queue.cc:260] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a516def1efa4b5cb3f92aa1e5935f7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39171 } } peers { permanent_uuid: "db631fac145841c7914b447fc8d98bd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41617 } } peers { permanent_uuid: "20db83031a35401c84c52f130e11c3c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41929 } }
I20260619 18:40:05.768116 13678 tablet_bootstrap.cc:492] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9: No bootstrap required, opened a new log
I20260619 18:40:05.768410 13678 ts_tablet_manager.cc:1397] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260619 18:40:05.769153 13676 ts_tablet_manager.cc:1428] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260619 18:40:05.769554 13374 heartbeater.cc:503] Master 127.0.98.62:44097 was elected leader, sending a full tablet report...
I20260619 18:40:05.773649 13678 raft_consensus.cc:348] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a516def1efa4b5cb3f92aa1e5935f7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39171 } } peers { permanent_uuid: "db631fac145841c7914b447fc8d98bd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41617 } } peers { permanent_uuid: "20db83031a35401c84c52f130e11c3c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41929 } }
I20260619 18:40:05.774031 13678 raft_consensus.cc:374] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:05.774096 13678 raft_consensus.cc:729] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db631fac145841c7914b447fc8d98bd9, State: Initialized, Role: FOLLOWER
I20260619 18:40:05.774593 13678 consensus_queue.cc:260] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a516def1efa4b5cb3f92aa1e5935f7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39171 } } peers { permanent_uuid: "db631fac145841c7914b447fc8d98bd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41617 } } peers { permanent_uuid: "20db83031a35401c84c52f130e11c3c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41929 } }
I20260619 18:40:05.775812 13678 ts_tablet_manager.cc:1428] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 18:40:05.775921 13642 heartbeater.cc:503] Master 127.0.98.62:44097 was elected leader, sending a full tablet report...
I20260619 18:40:05.855759 13682 raft_consensus.cc:482] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:05.856089 13682 raft_consensus.cc:504] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a516def1efa4b5cb3f92aa1e5935f7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39171 } } peers { permanent_uuid: "db631fac145841c7914b447fc8d98bd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41617 } } peers { permanent_uuid: "20db83031a35401c84c52f130e11c3c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41929 } }
I20260619 18:40:05.858948 13682 leader_election.cc:288] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a516def1efa4b5cb3f92aa1e5935f7e (127.0.98.2:39171), db631fac145841c7914b447fc8d98bd9 (127.0.98.3:41617)
I20260619 18:40:05.864413 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:05.858815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52905 (local address 127.0.98.3:41617)
0619 18:40:05.858918 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:05.858922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:05.859994 (+ 1072us) server_negotiation.cc:407] Connection header received
0619 18:40:05.860187 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:05.860191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:05.860240 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:05.860309 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:05.860708 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.861447 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.863061 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.863689 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.863766 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:05.863830 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:05.864175 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 18:40:05.864258 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:40:05.865075 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:05.858468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41617 (local address 127.0.98.1:52905)
0619 18:40:05.859885 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:05.859911 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:05.860067 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:05.860442 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:05.860445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:05.860462 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:05.860579 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:05.860585 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.861588 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.861591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:05.862919 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:05.862930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.863933 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.863937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:05.864034 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:05.864057 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:40:05.864824 (+ 767us) client_negotiation.cc:240] Negotiation successful
0619 18:40:05.864913 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1347,"mutex_wait_us":29,"thread_start_us":84,"threads_started":1}
I20260619 18:40:05.865839 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350a470ad70e484b9358ad5f7b9edbb8" candidate_uuid: "20db83031a35401c84c52f130e11c3c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db631fac145841c7914b447fc8d98bd9" is_pre_election: true
I20260619 18:40:05.866166 13591 raft_consensus.cc:2454] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20db83031a35401c84c52f130e11c3c9 in term 0.
I20260619 18:40:05.866664 13263 leader_election.cc:302] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20db83031a35401c84c52f130e11c3c9, db631fac145841c7914b447fc8d98bd9; no voters:
I20260619 18:40:05.867360 13682 raft_consensus.cc:2781] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:40:05.867451 13682 raft_consensus.cc:482] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:40:05.867488 13682 raft_consensus.cc:3037] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:05.868714 13682 raft_consensus.cc:504] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a516def1efa4b5cb3f92aa1e5935f7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39171 } } peers { permanent_uuid: "db631fac145841c7914b447fc8d98bd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41617 } } peers { permanent_uuid: "20db83031a35401c84c52f130e11c3c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41929 } }
I20260619 18:40:05.869065 13682 leader_election.cc:288] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [CANDIDATE]: Term 1 election: Requested vote from peers 1a516def1efa4b5cb3f92aa1e5935f7e (127.0.98.2:39171), db631fac145841c7914b447fc8d98bd9 (127.0.98.3:41617)
I20260619 18:40:05.869517 13591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350a470ad70e484b9358ad5f7b9edbb8" candidate_uuid: "20db83031a35401c84c52f130e11c3c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db631fac145841c7914b447fc8d98bd9"
I20260619 18:40:05.869643 13591 raft_consensus.cc:3037] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:05.869699 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:05.858066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50753 (local address 127.0.98.2:39171)
0619 18:40:05.858180 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:05.858185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:05.859376 (+ 1191us) server_negotiation.cc:407] Connection header received
0619 18:40:05.859572 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:05.859575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:05.859624 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:05.859668 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:05.860158 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.860987 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.866534 (+ 5547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.867136 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.867234 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:05.867290 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:05.869503 (+ 2213us) server_negotiation.cc:299] Negotiation successful
0619 18:40:05.869588 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:40:05.870118 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:05.857635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39171 (local address 127.0.98.1:50753)
0619 18:40:05.859269 (+ 1634us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:05.859283 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:05.859455 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:05.859818 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:05.859821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:05.859839 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:05.860035 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:05.860042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.861161 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.861164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:05.866393 (+ 5229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:05.866403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.869248 (+ 2845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.869251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:05.869348 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:05.869393 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 18:40:05.869878 (+ 485us) client_negotiation.cc:240] Negotiation successful
0619 18:40:05.869985 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1536,"mutex_wait_us":3920,"thread_start_us":140,"threads_started":1}
I20260619 18:40:05.870815 13463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350a470ad70e484b9358ad5f7b9edbb8" candidate_uuid: "20db83031a35401c84c52f130e11c3c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a516def1efa4b5cb3f92aa1e5935f7e" is_pre_election: true
I20260619 18:40:05.870982 13591 raft_consensus.cc:2454] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20db83031a35401c84c52f130e11c3c9 in term 1.
I20260619 18:40:05.871084 13463 raft_consensus.cc:2454] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20db83031a35401c84c52f130e11c3c9 in term 0.
I20260619 18:40:05.871359 13462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "350a470ad70e484b9358ad5f7b9edbb8" candidate_uuid: "20db83031a35401c84c52f130e11c3c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a516def1efa4b5cb3f92aa1e5935f7e"
I20260619 18:40:05.871474 13462 raft_consensus.cc:3037] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:05.871559 13263 leader_election.cc:302] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20db83031a35401c84c52f130e11c3c9, db631fac145841c7914b447fc8d98bd9; no voters:
I20260619 18:40:05.872561 13462 raft_consensus.cc:2454] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20db83031a35401c84c52f130e11c3c9 in term 1.
I20260619 18:40:05.873238 13682 raft_consensus.cc:2781] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:05.873449 13682 raft_consensus.cc:686] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [term 1 LEADER]: Becoming Leader. State: Replica: 20db83031a35401c84c52f130e11c3c9, State: Running, Role: LEADER
I20260619 18:40:05.873772 13682 consensus_queue.cc:237] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a516def1efa4b5cb3f92aa1e5935f7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39171 } } peers { permanent_uuid: "db631fac145841c7914b447fc8d98bd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41617 } } peers { permanent_uuid: "20db83031a35401c84c52f130e11c3c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41929 } }
I20260619 18:40:05.877110 13175 catalog_manager.cc:5495] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20db83031a35401c84c52f130e11c3c9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "20db83031a35401c84c52f130e11c3c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a516def1efa4b5cb3f92aa1e5935f7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db631fac145841c7914b447fc8d98bd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20db83031a35401c84c52f130e11c3c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41929 } health_report { overall_health: HEALTHY } } }
Jun 19 18:40:05 dist-test-slave-rxv6 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260619 18:40:05.893954 13375 tablet.cc:2369] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:40:05.905278 13510 tablet.cc:2369] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:40:05.908010 13643 tablet.cc:2369] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:40:05.932255 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:05.926644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33456 (local address 127.0.98.1:41929)
0619 18:40:05.927114 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:05.927118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:05.927131 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:40:05.927218 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:05.927221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:05.927286 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:05.927395 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:05.927719 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.928531 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.930112 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:05.930468 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:05.930571 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:05.930940 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:05.931492 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:05.931605 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:05.932024 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 18:40:05.932093 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379}
I20260619 18:40:05.939635 13591 raft_consensus.cc:1261] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9 [term 1 FOLLOWER]: Refusing update from remote peer 20db83031a35401c84c52f130e11c3c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:05.940529 13682 consensus_queue.cc:1035] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db631fac145841c7914b447fc8d98bd9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:05.945642 13462 raft_consensus.cc:1261] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e [term 1 FOLLOWER]: Refusing update from remote peer 20db83031a35401c84c52f130e11c3c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:05.947432 13687 consensus_queue.cc:1035] T 350a470ad70e484b9358ad5f7b9edbb8 P 20db83031a35401c84c52f130e11c3c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a516def1efa4b5cb3f92aa1e5935f7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:05.961807 13690 mvcc.cc:205] Tried to move back new op lower bound from 7298639486720606208 to 7298639486462164992. Current Snapshot: MvccSnapshot[applied={T|T < 7298639486720606208}]
I20260619 18:40:05.963716 13692 mvcc.cc:205] Tried to move back new op lower bound from 7298639486720606208 to 7298639486462164992. Current Snapshot: MvccSnapshot[applied={T|T < 7298639486720606208}]
I20260619 18:40:05.968954 13694 mvcc.cc:205] Tried to move back new op lower bound from 7298639486720606208 to 7298639486462164992. Current Snapshot: MvccSnapshot[applied={T|T < 7298639486720606208}]
I20260619 18:40:05.978893 13175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:05.979091 13175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:05.981845 13175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1721f19404e74044960542637d56c74b: Sending DeleteTablet for 3 replicas of tablet 350a470ad70e484b9358ad5f7b9edbb8
I20260619 18:40:05.983162 13443 tablet_service.cc:1514] Processing DeleteTablet for tablet 350a470ad70e484b9358ad5f7b9edbb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45068
I20260619 18:40:05.983664 13576 tablet_service.cc:1514] Processing DeleteTablet for tablet 350a470ad70e484b9358ad5f7b9edbb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36092
I20260619 18:40:05.984730 13308 tablet_service.cc:1514] Processing DeleteTablet for tablet 350a470ad70e484b9358ad5f7b9edbb8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33444
I20260619 18:40:05.985754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 13213
I20260619 18:40:05.986122 13700 tablet_replica.cc:330] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e: stopping tablet replica
I20260619 18:40:05.986490 13700 raft_consensus.cc:2229] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:05.986845 13700 raft_consensus.cc:2258] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:05.994128 13701 tablet_replica.cc:330] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9: stopping tablet replica
I20260619 18:40:05.994588 13701 raft_consensus.cc:2229] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:05.995235 13701 raft_consensus.cc:2258] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:05.996470 13700 ts_tablet_manager.cc:1905] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:06.000128 13700 ts_tablet_manager.cc:1918] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:40:06.000345 13700 log.cc:1196] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/350a470ad70e484b9358ad5f7b9edbb8
I20260619 18:40:06.000792 13700 ts_tablet_manager.cc:1939] T 350a470ad70e484b9358ad5f7b9edbb8 P 1a516def1efa4b5cb3f92aa1e5935f7e: Deleting consensus metadata
W20260619 18:40:06.001835 13159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41929 (error 108)
I20260619 18:40:06.002434 13161 catalog_manager.cc:4841] TS 1a516def1efa4b5cb3f92aa1e5935f7e (127.0.98.2:39171): tablet 350a470ad70e484b9358ad5f7b9edbb8 (table test-table [id=c76bd0ad5dd34e67901c101afe5fee44]) successfully deleted
I20260619 18:40:06.002658 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 13378
I20260619 18:40:06.004808 13701 ts_tablet_manager.cc:1905] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:06.007661 13701 ts_tablet_manager.cc:1918] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:40:06.007762 13701 log.cc:1196] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/350a470ad70e484b9358ad5f7b9edbb8
I20260619 18:40:06.008071 13701 ts_tablet_manager.cc:1939] T 350a470ad70e484b9358ad5f7b9edbb8 P db631fac145841c7914b447fc8d98bd9: Deleting consensus metadata
I20260619 18:40:06.009620 13161 catalog_manager.cc:4841] TS db631fac145841c7914b447fc8d98bd9 (127.0.98.3:41617): tablet 350a470ad70e484b9358ad5f7b9edbb8 (table test-table [id=c76bd0ad5dd34e67901c101afe5fee44]) successfully deleted
I20260619 18:40:06.010478 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:06.010134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41929 (local address 127.0.0.1:33470)
0619 18:40:06.010309 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:06.010387 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41929: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260619 18:40:06.010744 13159 catalog_manager.cc:4584] TS 20db83031a35401c84c52f130e11c3c9 (127.0.98.1:41929): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 350a470ad70e484b9358ad5f7b9edbb8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41929: connect: Connection refused (error 111)
I20260619 18:40:06.013048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 13511
I20260619 18:40:06.021036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 13142
I20260619 18:40:06.028764 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 18:40:06.363289 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 18:40:06.363427 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 18:40:06.698657 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 18:40:06.698 UTC [12209] LOG: received smart shutdown request
2026-06-19 18:40:06.702 UTC [12209] LOG: background worker "logical replication launcher" (PID 12218) exited with exit code 1
2026-06-19 18:40:06.702 UTC [12213] LOG: shutting down
2026-06-19 18:40:06.864 UTC [12209] LOG: database system is shut down
2026-06-19T18:40:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47767 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:40:07 dist-test-slave-rxv6 krb5kdc[13712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:40:07 dist-test-slave-rxv6 krb5kdc[13712](info): set up 2 sockets
Jun 19 18:40:07 dist-test-slave-rxv6 krb5kdc[13712](info): commencing operation
krb5kdc: starting...
W20260619 18:40:07.933853 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:40:08 dist-test-slave-rxv6 krb5kdc[13712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:40:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:40:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:40:08.046979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34003
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34003,127.0.98.61:42001,127.0.98.60:45647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:08.155018 13725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:08.155306 13725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:08.155372 13725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:08.159369 13725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:40:08.159431 13725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:08.159458 13725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:08.159479 13725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:40:08.159510 13725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:40:08.164827 13725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34003,127.0.98.61:42001,127.0.98.60:45647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:08.165892 13725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:08.166901 13725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:08.174857 13731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:08.174993 13725 server_base.cc:1034] running on GCE node
W20260619 18:40:08.174846 13733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:08.174887 13730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:08.175726 13725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:08.176748 13725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:08.177935 13725 hybrid_clock.cc:648] HybridClock initialized: now 1781894408177896 us; error 26 us; skew 500 ppm
Jun 19 18:40:08 dist-test-slave-rxv6 krb5kdc[13712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:08.184127 13725 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:40:08.185004 13725 webserver.cc:460] Webserver started at http://127.0.98.62:35897/ using document root <none> and password file <none>
I20260619 18:40:08.185537 13725 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:08.185602 13725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:08.185799 13725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:08.187894 13725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "ee53c95cd3354facb37b89c399d3ccc6"
format_stamp: "Formatted at 2026-06-19 18:40:08 on dist-test-slave-rxv6"
server_key: "1e36b04c5ecbd5b83430f70df3b5e91d"
server_key_iv: "1df79fe84d6e2b1ae0f324e19f6e86c3"
server_key_version: "encryptionkey@0"
I20260619 18:40:08.188431 13725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee53c95cd3354facb37b89c399d3ccc6"
format_stamp: "Formatted at 2026-06-19 18:40:08 on dist-test-slave-rxv6"
server_key: "1e36b04c5ecbd5b83430f70df3b5e91d"
server_key_iv: "1df79fe84d6e2b1ae0f324e19f6e86c3"
server_key_version: "encryptionkey@0"
I20260619 18:40:08.191987 13725 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:40:08.194310 13740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:08.195509 13725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:40:08.195619 13725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "ee53c95cd3354facb37b89c399d3ccc6"
format_stamp: "Formatted at 2026-06-19 18:40:08 on dist-test-slave-rxv6"
server_key: "1e36b04c5ecbd5b83430f70df3b5e91d"
server_key_iv: "1df79fe84d6e2b1ae0f324e19f6e86c3"
server_key_version: "encryptionkey@0"
I20260619 18:40:08.195724 13725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:08.211833 13725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:08.214846 13725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:08.215013 13725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:08.221819 13725 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34003
I20260619 18:40:08.221832 13792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34003 every 8 connection(s)
I20260619 18:40:08.222750 13725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:40:08.225476 13793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:08.229161 13793 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:40:08.233000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 13725
I20260619 18:40:08.233192 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:40:08.233506 392 external_mini_cluster.cc:1428] Setting key 341c9a6674e1ff921e1add27d99fc337
Jun 19 18:40:08 dist-test-slave-rxv6 krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260619 18:40:08.244351 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.231349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36770 (local address 127.0.98.62:34003)
0619 18:40:08.231780 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:08.231793 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:08.231960 (+ 167us) server_negotiation.cc:407] Connection header received
0619 18:40:08.233013 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:08.233018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:08.233396 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:08.233606 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:08.234632 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.235490 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.236667 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.237195 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.237363 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.239666 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:08.239673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:08.239686 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:08.239712 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:08.241518 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.241949 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.241954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.241957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.242059 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.242387 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.242390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.242391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.242764 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:08.242875 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:08.243226 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 18:40:08.243401 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"spinlock_wait_cycles":9472,"thread_start_us":149,"threads_started":1}
I20260619 18:40:08.244357 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.231349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34003 (local address 127.0.0.1:36770)
0619 18:40:08.231808 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.231855 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:08.232688 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:08.233603 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:08.233608 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:08.234112 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:08.234455 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.234472 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.235617 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.235620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.236530 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.236540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.237344 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.237348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.237465 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.238128 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:08.238145 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:08.239507 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:08.241632 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.241640 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.241647 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.241843 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.242150 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.242153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.242155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.242286 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.242873 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:08.242878 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:08.243003 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 18:40:08.243166 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 18:40:08.243525 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":89,"thread_start_us":107,"threads_started":1}
I20260619 18:40:08.246755 13793 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:40:08.247416 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.247186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42001 (local address 127.0.0.1:38572)
0619 18:40:08.247294 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.247360 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 18:40:08.247573 13741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42001: connect: Connection refused (error 111)
I20260619 18:40:08.248378 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.248192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42001 (local address 127.0.0.1:38574)
0619 18:40:08.248285 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.248327 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 18:40:08.248493 13793 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42001: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42001: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 18:40:08.258874 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42001
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34003,127.0.98.61:42001,127.0.98.60:45647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:40:08.297747 13793 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } attempt: 1
I20260619 18:40:08.298718 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.298352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42001 (local address 127.0.0.1:38576)
0619 18:40:08.298535 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.298617 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 18:40:08.299496 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.299238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42001 (local address 127.0.0.1:38592)
0619 18:40:08.299369 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.299425 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 18:40:08.299623 13793 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42001: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42001: connect: Connection refused (error 111)
W20260619 18:40:08.366312 13800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:08.366565 13800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:08.366603 13800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 18:40:08.367789 13793 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } attempt: 2
I20260619 18:40:08.368569 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.368268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42001 (local address 127.0.0.1:38606)
0619 18:40:08.368418 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.368483 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 18:40:08.369238 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.369033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42001 (local address 127.0.0.1:38618)
0619 18:40:08.369121 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.369168 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 18:40:08.369355 13793 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42001: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42001: connect: Connection refused (error 111)
W20260619 18:40:08.370896 13800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:40:08.370942 13800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:08.370960 13800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:08.370975 13800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:40:08.370993 13800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:40:08.376238 13800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34003,127.0.98.61:42001,127.0.98.60:45647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:08.377161 13800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:08.378091 13800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:08.386039 13806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:08.386080 13800 server_base.cc:1034] running on GCE node
W20260619 18:40:08.386097 13808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:08.386039 13805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:08.386732 13800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:08.387744 13800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:08.388904 13800 hybrid_clock.cc:648] HybridClock initialized: now 1781894408388887 us; error 28 us; skew 500 ppm
Jun 19 18:40:08 dist-test-slave-rxv6 krb5kdc[13712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:08.391480 13800 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 18:40:08.392320 13800 webserver.cc:460] Webserver started at http://127.0.98.61:43267/ using document root <none> and password file <none>
I20260619 18:40:08.392840 13800 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:08.392906 13800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:08.393110 13800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:08.395174 13800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-1/data/instance:
uuid: "66a53bf865d344728cad887eeb8b6920"
format_stamp: "Formatted at 2026-06-19 18:40:08 on dist-test-slave-rxv6"
server_key: "ee237dfe01cd0009d8922c08aef8a4e9"
server_key_iv: "da57db4d0433801d94c6b03ca3fc5e8f"
server_key_version: "encryptionkey@0"
I20260619 18:40:08.395695 13800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-1/wal/instance:
uuid: "66a53bf865d344728cad887eeb8b6920"
format_stamp: "Formatted at 2026-06-19 18:40:08 on dist-test-slave-rxv6"
server_key: "ee237dfe01cd0009d8922c08aef8a4e9"
server_key_iv: "da57db4d0433801d94c6b03ca3fc5e8f"
server_key_version: "encryptionkey@0"
I20260619 18:40:08.399181 13800 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:40:08.401551 13815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:08.402732 13800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:40:08.402848 13800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-1/wal
uuid: "66a53bf865d344728cad887eeb8b6920"
format_stamp: "Formatted at 2026-06-19 18:40:08 on dist-test-slave-rxv6"
server_key: "ee237dfe01cd0009d8922c08aef8a4e9"
server_key_iv: "da57db4d0433801d94c6b03ca3fc5e8f"
server_key_version: "encryptionkey@0"
I20260619 18:40:08.402935 13800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:08.425107 13800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:08.428128 13800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:08.428315 13800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:08.435038 13800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42001
I20260619 18:40:08.435055 13867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42001 every 8 connection(s)
I20260619 18:40:08.435968 13800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-1/data/info.pb
I20260619 18:40:08.438845 13868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:08.442646 13868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:40:08.444871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 13800
I20260619 18:40:08.444972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-1/wal/instance
I20260619 18:40:08.445209 392 external_mini_cluster.cc:1428] Setting key c40957d42be72a23f2b8062284d28ec3
Jun 19 18:40:08 dist-test-slave-rxv6 krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260619 18:40:08.456382 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.444532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36776 (local address 127.0.98.62:34003)
0619 18:40:08.444703 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:08.444708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:08.445512 (+ 804us) server_negotiation.cc:407] Connection header received
0619 18:40:08.446489 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:08.446492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:08.446548 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:08.446641 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:08.447843 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.448378 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.450009 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.450252 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.450337 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.452973 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:08.452976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:08.452978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:08.453001 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:08.454351 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.454846 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.454849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.454850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.454919 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.455298 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.455301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.455303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.455498 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:08.455573 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:08.456150 (+ 577us) server_negotiation.cc:299] Negotiation successful
0619 18:40:08.456265 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:40:08.457261 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.444864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34003 (local address 127.0.0.1:36776)
0619 18:40:08.445353 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.445404 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:08.446284 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:08.446800 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:08.446807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:08.447144 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:08.447701 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.447718 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.448497 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.448501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.449864 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.449876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.450403 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.450407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.450610 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.451190 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:08.451209 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:08.452815 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:08.454448 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.454452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.454463 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.454760 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.454991 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.454993 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.454994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.455144 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.455595 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:08.455601 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:08.455899 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 18:40:08.456096 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 18:40:08.456442 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":123,"threads_started":1}
I20260619 18:40:08.458595 13868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:40:08.460494 13793 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } attempt: 3
Jun 19 18:40:08 dist-test-slave-rxv6 krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 18:40:08 dist-test-slave-rxv6 krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 18:40:08.468345 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.459034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42001 (local address 127.0.0.1:38632)
0619 18:40:08.459182 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.459193 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:08.459249 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:08.459781 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:08.459784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:08.459945 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:08.460014 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.460019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.461070 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.461073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.461797 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.461805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.462358 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.462361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.462441 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.462866 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:08.462880 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:08.464914 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:08.466942 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.466945 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.466947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.467114 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.467366 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.467369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.467371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.467443 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.467838 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:08.467841 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:08.467920 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 18:40:08.468009 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 18:40:08.468164 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 18:40:08.469022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45647
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34003,127.0.98.61:42001,127.0.98.60:45647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:40:08.469725 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.460880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42001 (local address 127.0.0.1:38648)
0619 18:40:08.460994 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.461007 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:08.461070 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:08.461573 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:08.461577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:08.461731 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:08.462322 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.462332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.463086 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.463089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.463785 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.463795 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.464324 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.464328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.464439 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.464860 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:08.464871 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:08.466078 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:08.468334 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.468338 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.468340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.468556 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.468824 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.468827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.468829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.468908 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.469260 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:08.469263 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:08.469355 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 18:40:08.469437 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 18:40:08.469563 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 18:40:08.469748 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.461051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.0.98.61:42001)
0619 18:40:08.461355 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:08.461368 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:08.461380 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:40:08.461410 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:08.461413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:08.461448 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:08.461529 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:08.462461 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.462974 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.463901 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.464213 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.464307 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.466214 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:08.466232 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:08.466234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:08.466254 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:08.468220 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.468655 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.468658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.468659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.468752 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.468985 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.468988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.468990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.469146 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:08.469262 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:08.469495 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 18:40:08.469587 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":123,"threads_started":1}
I20260619 18:40:08.468377 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.459135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.0.98.61:42001)
0619 18:40:08.459471 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:08.459479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:08.459495 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:40:08.459527 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:08.459531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:08.459676 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:08.459787 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:08.460105 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.460963 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.461887 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.462264 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.462352 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.465027 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:08.465037 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:08.465050 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:08.465073 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:08.466850 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.467208 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.467214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.467216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.467291 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.467514 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.467517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.467519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.467746 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:08.467860 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:08.468096 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 18:40:08.468226 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":172,"threads_started":1}
I20260619 18:40:08.471478 13868 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:40:08.471589 13793 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:40:08.472165 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.471916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45647 (local address 127.0.0.1:55844)
0619 18:40:08.472028 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.472103 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 18:40:08.472186 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.471930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45647 (local address 127.0.0.1:55846)
0619 18:40:08.472087 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.472128 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260619 18:40:08.472440 13816 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
I20260619 18:40:08.472784 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.472532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45647 (local address 127.0.0.1:55860)
0619 18:40:08.472687 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.472724 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260619 18:40:08.472981 13793 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45647: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
I20260619 18:40:08.473263 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.473088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45647 (local address 127.0.0.1:55874)
0619 18:40:08.473171 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.473218 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 18:40:08.473382 13868 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45647: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
I20260619 18:40:08.504137 13793 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } attempt: 1
I20260619 18:40:08.504956 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.504675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45647 (local address 127.0.0.1:55890)
0619 18:40:08.504818 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.504877 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 18:40:08.505591 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.505385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45647 (local address 127.0.0.1:55906)
0619 18:40:08.505482 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.505525 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 18:40:08.505710 13793 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45647: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
I20260619 18:40:08.522634 13868 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } attempt: 1
I20260619 18:40:08.523542 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.523213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45647 (local address 127.0.0.1:55922)
0619 18:40:08.523394 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.523463 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 18:40:08.524251 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.524048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45647 (local address 127.0.0.1:55936)
0619 18:40:08.524146 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.524197 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 18:40:08.524354 13868 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45647: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
W20260619 18:40:08.578275 13875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:08.578537 13875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:08.578593 13875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 18:40:08.580868 13793 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } attempt: 2
I20260619 18:40:08.581722 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.581410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45647 (local address 127.0.0.1:55942)
0619 18:40:08.581565 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.581631 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 18:40:08.582450 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.582242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45647 (local address 127.0.0.1:55958)
0619 18:40:08.582351 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.582394 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 18:40:08.582608 13793 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45647: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
W20260619 18:40:08.582911 13875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:40:08.582964 13875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:08.582983 13875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:08.582998 13875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:40:08.583015 13875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:40:08.588328 13875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34003,127.0.98.61:42001,127.0.98.60:45647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:08.589352 13875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:08.590390 13875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:40:08.592510 13868 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } attempt: 2
I20260619 18:40:08.593261 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.593009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45647 (local address 127.0.0.1:55964)
0619 18:40:08.593132 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.593190 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 18:40:08.593986 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.593748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45647 (local address 127.0.0.1:55980)
0619 18:40:08.593848 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.593894 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 18:40:08.594106 13868 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45647: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45647: connect: Connection refused (error 111)
W20260619 18:40:08.598765 13881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:08.598843 13884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:08.598834 13875 server_base.cc:1034] running on GCE node
W20260619 18:40:08.598762 13882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:08.599493 13875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:08.600574 13875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:08.601748 13875 hybrid_clock.cc:648] HybridClock initialized: now 1781894408601721 us; error 38 us; skew 500 ppm
Jun 19 18:40:08 dist-test-slave-rxv6 krb5kdc[13712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:08.605005 13875 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 18:40:08.605984 13875 webserver.cc:460] Webserver started at http://127.0.98.60:44169/ using document root <none> and password file <none>
I20260619 18:40:08.606571 13875 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:08.606635 13875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:08.606827 13875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:08.608785 13875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-2/data/instance:
uuid: "3f03a4bfc87b41e282a342b60f6a22d0"
format_stamp: "Formatted at 2026-06-19 18:40:08 on dist-test-slave-rxv6"
server_key: "2039e60db0a7ba4e64464611da07d8ee"
server_key_iv: "98674bcd8e30c726daf07f9a58af8f0e"
server_key_version: "encryptionkey@0"
I20260619 18:40:08.609300 13875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-2/wal/instance:
uuid: "3f03a4bfc87b41e282a342b60f6a22d0"
format_stamp: "Formatted at 2026-06-19 18:40:08 on dist-test-slave-rxv6"
server_key: "2039e60db0a7ba4e64464611da07d8ee"
server_key_iv: "98674bcd8e30c726daf07f9a58af8f0e"
server_key_version: "encryptionkey@0"
I20260619 18:40:08.612875 13875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:40:08.615307 13891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:08.616442 13875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:40:08.616556 13875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-2/wal
uuid: "3f03a4bfc87b41e282a342b60f6a22d0"
format_stamp: "Formatted at 2026-06-19 18:40:08 on dist-test-slave-rxv6"
server_key: "2039e60db0a7ba4e64464611da07d8ee"
server_key_iv: "98674bcd8e30c726daf07f9a58af8f0e"
server_key_version: "encryptionkey@0"
I20260619 18:40:08.616650 13875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:08.633793 13875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:08.636610 13875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:08.636782 13875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:08.643404 13875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45647
I20260619 18:40:08.643422 13943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45647 every 8 connection(s)
I20260619 18:40:08.644309 13875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-2/data/info.pb
I20260619 18:40:08.645406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 13875
I20260619 18:40:08.645509 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781894249862417-392-0/minicluster-data/master-2/wal/instance
I20260619 18:40:08.645828 392 external_mini_cluster.cc:1428] Setting key 0a13cc279a8d90644e6c6c3bf02df2c4
I20260619 18:40:08.647307 13944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:08.651506 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:40:08 dist-test-slave-rxv6 krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 18:40:08 dist-test-slave-rxv6 krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:08.660044 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.647230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55988 (local address 127.0.98.60:45647)
0619 18:40:08.647610 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:08.647627 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:08.647661 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:40:08.648299 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:08.648314 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:08.648629 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:08.648901 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:08.649801 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.650822 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.651939 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.652403 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.652518 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.655243 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:08.655248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:08.655257 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:08.655293 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:08.657010 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.657479 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.657485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.657490 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.657617 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.657900 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.657903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.657924 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.658312 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:08.658517 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:08.658776 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 18:40:08.659053 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":124,"threads_started":1}
I20260619 18:40:08.662087 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.653375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36778 (local address 127.0.98.62:34003)
0619 18:40:08.653531 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:08.653536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:08.653775 (+ 239us) server_negotiation.cc:407] Connection header received
0619 18:40:08.653851 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:08.653854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:08.653934 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:08.654016 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:08.654814 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.655318 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.656455 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.656674 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.656770 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.658991 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:08.658995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:08.658998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:08.659025 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:08.660435 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.660836 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.660839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.660840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.660926 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.661178 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.661181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.661182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.661373 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:08.661469 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:08.661824 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 18:40:08.661966 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 18:40:08.662098 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.653290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34003 (local address 127.0.0.1:36778)
0619 18:40:08.653605 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.653651 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:08.653761 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:08.654050 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:08.654054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:08.654506 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:08.654666 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.654673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.655432 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.655435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.656336 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.656348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.656793 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.656797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.656910 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.657390 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:08.657401 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:08.658823 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:08.660528 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.660535 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.660539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.660749 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.661004 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.661007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.661008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.661093 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.661480 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:08.661484 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:08.661608 (+ 124us) client_negotiation.cc:769] Sending connection context
0619 18:40:08.661738 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 18:40:08.661891 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":81,"threads_started":1}
I20260619 18:40:08.662986 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:40:08 dist-test-slave-rxv6 krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 18:40:08 dist-test-slave-rxv6 krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 18:40:08.670585 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.662542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.0.98.62:34003)
0619 18:40:08.662691 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:08.662695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:08.662706 (+ 11us) server_negotiation.cc:407] Connection header received
0619 18:40:08.662734 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:08.662737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:08.662771 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:08.662830 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:08.663408 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.663884 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.664747 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.664956 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.665030 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.667477 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:08.667480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:08.667482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:08.667503 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:08.668869 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.669305 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.669309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.669311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.669415 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.669717 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.669720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.669722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.669971 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:08.670210 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:08.670315 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 18:40:08.670428 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 18:40:08.672003 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.663515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38658 (local address 127.0.98.61:42001)
0619 18:40:08.663621 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:08.663625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:08.664082 (+ 457us) server_negotiation.cc:407] Connection header received
0619 18:40:08.664149 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:08.664152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:08.664220 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:08.664299 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:08.664679 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.665306 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.666173 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.666548 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.666633 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.668634 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:08.668638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:08.668640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:08.668669 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:08.670307 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.670790 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.670794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.670795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.670884 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.671154 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.671156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.671158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.671388 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:08.671472 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:08.671756 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 18:40:08.671858 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 18:40:08.672421 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.663409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42001 (local address 127.0.0.1:38658)
0619 18:40:08.664023 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.664033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:08.664090 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:08.664348 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:08.664351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:08.664487 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:08.664575 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.664579 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.665427 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.665430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.666062 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.666071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.666670 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.666675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.666771 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.667188 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:08.667202 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:08.668486 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:08.670422 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.670425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.670427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.670642 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.670974 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.670977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.670979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.671071 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.671486 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:08.671489 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:08.671577 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 18:40:08.672129 (+ 552us) client_negotiation.cc:240] Negotiation successful
0619 18:40:08.672268 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554}
I20260619 18:40:08.673154 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:40:08 dist-test-slave-rxv6 krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 18:40:08 dist-test-slave-rxv6 krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 18:40:08.681623 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.673609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45647 (local address 127.0.0.1:55994)
0619 18:40:08.673697 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.673710 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:08.673787 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:08.674093 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:08.674096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:08.674253 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:08.674339 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.674345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.675563 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.675566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.676055 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.676063 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.676514 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.676517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.676588 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.676993 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:08.677003 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:08.678501 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:08.680057 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.680061 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.680062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.680273 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.680550 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.680552 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.680554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.680668 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.681108 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:08.681111 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:08.681201 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 18:40:08.681307 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 18:40:08.681448 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 18:40:08.681658 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.671826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.0.98.61:42001)
0619 18:40:08.671939 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:08.671944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:08.671956 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:40:08.671989 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:08.671992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:08.672030 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:08.672099 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:08.672942 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.673624 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.674610 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.674974 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.675076 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.678219 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:08.678222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:08.678224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:08.678249 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:08.679911 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.680368 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.680371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.680373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.680468 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.680785 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.680789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.680791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.681009 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:08.681233 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:08.681373 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 18:40:08.681494 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:40:08.681721 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.673765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.0.98.60:45647)
0619 18:40:08.673876 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:08.673880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:08.673889 (+ 9us) server_negotiation.cc:407] Connection header received
0619 18:40:08.673965 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:08.673967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:08.674004 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:08.674062 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:08.674444 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.675464 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.676173 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.676402 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.676502 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.678624 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:08.678627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:08.678629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:08.678650 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:08.679958 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.680368 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.680371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.680373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.680468 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.680785 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.680789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.680791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.681009 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:08.681106 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:08.681373 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 18:40:08.681510 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":39}
I20260619 18:40:08.681972 13793 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } attempt: 3
I20260619 18:40:08.685477 13868 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } attempt: 3
I20260619 18:40:08.685806 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0: Bootstrap starting.
I20260619 18:40:08.685860 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Jun 19 18:40:08 dist-test-slave-rxv6 krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 18:40:08.688814 13944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:08.689951 13944 log.cc:824] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:08.690420 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.682412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45647 (local address 127.0.0.1:56006)
0619 18:40:08.682522 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.682536 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:08.682612 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:08.682869 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:08.682872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:08.683029 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:08.683130 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.683138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.683883 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.683886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.684584 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.684593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.685000 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.685003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.685084 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.685510 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:08.685523 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:08.687088 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:08.688897 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.688900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.688903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.689138 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.689424 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.689427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.689429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.689512 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.689929 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:08.689932 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:08.690029 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 18:40:08.690119 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 18:40:08.690258 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 18:40:08.691166 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.682584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56006 (local address 127.0.98.60:45647)
0619 18:40:08.682686 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:08.682690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:08.682702 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:40:08.682731 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:08.682733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:08.682770 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:08.682841 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:08.683251 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.683790 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.684690 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.684910 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.684988 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.687221 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:08.687225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:08.687228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:08.687250 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:08.688785 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.689244 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.689248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.689250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.689328 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.689599 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.689601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.689603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.689778 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:08.689861 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:08.690931 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0619 18:40:08.691033 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 18:40:08 dist-test-slave-rxv6 krb5kdc[13712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 18:40:08.692384 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0: No bootstrap required, opened a new log
I20260619 18:40:08.694782 13793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6: Bootstrap starting.
I20260619 18:40:08.696414 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.685989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45647 (local address 127.0.0.1:56020)
0619 18:40:08.686149 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:08.686162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:08.686230 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:08.687193 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:08.687195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:08.687364 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:08.687739 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.687750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.689055 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.689058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.689796 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:08.689805 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.690386 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.690389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:08.690467 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.690894 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:08.690907 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:08.692303 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:08.694477 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.694480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.694482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.694720 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.695015 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:08.695018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:08.695020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:08.695101 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:08.695503 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:08.695506 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:08.695828 (+ 322us) client_negotiation.cc:769] Sending connection context
0619 18:40:08.695910 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 18:40:08.696262 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 18:40:08.697007 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.686660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.0.98.60:45647)
0619 18:40:08.686957 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:08.686965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:08.686977 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:40:08.687008 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:08.687011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:08.687051 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:08.687852 (+ 801us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:08.687943 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.688941 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.689947 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.690269 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.692418 (+ 2149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.692517 (+ 99us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:08.692520 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:08.692523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:08.692539 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:08.694362 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.694836 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.694839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.694841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.694918 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.695207 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.695210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.695212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.695400 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:08.696655 (+ 1255us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:08.696746 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 18:40:08.696855 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":102,"threads_started":1}
I20260619 18:40:08.697018 13944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } }
I20260619 18:40:08.697363 13944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:08.697425 13944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f03a4bfc87b41e282a342b60f6a22d0, State: Initialized, Role: FOLLOWER
I20260619 18:40:08.697942 13944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } }
I20260619 18:40:08.699105 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.688378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.0.98.62:34003)
0619 18:40:08.688483 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:08.688486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:08.690182 (+ 1696us) server_negotiation.cc:407] Connection header received
0619 18:40:08.690355 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:08.690358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:08.690417 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:08.690468 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:08.691359 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.692021 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.693096 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.693418 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.693491 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.695279 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:08.695282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:08.695285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:08.695309 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:08.696800 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.697795 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.697798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.697800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.697898 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.698217 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.698220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.698222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.698420 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:08.698515 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:08.698832 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 18:40:08.698961 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:40:08.699330 13944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:40:08.700424 13868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920: Bootstrap starting.
I20260619 18:40:08.703311 13868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:08.704005 13868 log.cc:824] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:08.704865 13793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:08.705544 13793 log.cc:824] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:08.706194 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.689580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.0.98.60:45647)
0619 18:40:08.690916 (+ 1336us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:08.690924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:08.697140 (+ 6216us) server_negotiation.cc:407] Connection header received
0619 18:40:08.697180 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:08.697183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:08.697226 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:08.697290 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:08.698005 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.698781 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.699791 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.700096 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.700186 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.702147 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:08.702151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:08.702154 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:08.702178 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:08.703769 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.704554 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.704558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.704559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.704642 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.704923 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.704926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.704928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.705138 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:08.705268 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:08.705840 (+ 572us) server_negotiation.cc:299] Negotiation successful
0619 18:40:08.705992 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1257,"thread_start_us":105,"threads_started":1}
I20260619 18:40:08.707147 13868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920: No bootstrap required, opened a new log
I20260619 18:40:08.704520 13875 master_runner.cc:186] Error getting master registration for 127.0.98.62:34003: OK, instance_id { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" instance_seqno: 1781894408219867 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:40:08.707659 13875 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 18:40:08.710269 13970 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:40:08.710980 13970 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:40:08.712190 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:08.689077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38674 (local address 127.0.98.61:42001)
0619 18:40:08.689170 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:08.689173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:08.694169 (+ 4996us) server_negotiation.cc:407] Connection header received
0619 18:40:08.694273 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:08.694276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:08.694319 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:08.694388 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:08.700699 (+ 6311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.701447 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.703186 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:08.703524 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:08.703604 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:08.708750 (+ 5146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:08.708754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:08.708757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:08.708783 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:08.710477 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.710958 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.710961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.710963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.711053 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:08.711345 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:08.711349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:08.711351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:08.711546 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:08.711664 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:08.711924 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 18:40:08.712038 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 18:40:08.712469 13793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6: No bootstrap required, opened a new log
I20260619 18:40:08.714015 13868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } }
I20260619 18:40:08.714321 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } } }
I20260619 18:40:08.714589 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:40:08.717062 13793 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } }
I20260619 18:40:08.717303 13793 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:08.717363 13793 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee53c95cd3354facb37b89c399d3ccc6, State: Initialized, Role: FOLLOWER
I20260619 18:40:08.717991 13868 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:08.717900 13793 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } }
I20260619 18:40:08.718067 13868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a53bf865d344728cad887eeb8b6920, State: Initialized, Role: FOLLOWER
I20260619 18:40:08.718462 13974 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } } }
I20260619 18:40:08.718566 13868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } }
I20260619 18:40:08.718658 13974 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:40:08.719385 13793 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:40:08.719388 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } } }
I20260619 18:40:08.719585 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:40:08.720034 13868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 18:40:08.726348 13997 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:40:08.726518 13997 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:40:08.728080 13725 master_runner.cc:263] No leader master found from master ee53c95cd3354facb37b89c399d3ccc6
I20260619 18:40:08.728152 13725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:40:08.729668 13800 master_runner.cc:263] No leader master found from master 66a53bf865d344728cad887eeb8b6920
I20260619 18:40:08.729732 13800 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 18:40:08.731007 13991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:40:08.731091 13991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:40:08.838186 13974 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:08.838424 13974 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } }
I20260619 18:40:08.839587 13974 leader_election.cc:288] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66a53bf865d344728cad887eeb8b6920 (127.0.98.61:42001), 3f03a4bfc87b41e282a342b60f6a22d0 (127.0.98.60:45647)
I20260619 18:40:08.840581 13841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee53c95cd3354facb37b89c399d3ccc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a53bf865d344728cad887eeb8b6920" is_pre_election: true
I20260619 18:40:08.840972 13841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee53c95cd3354facb37b89c399d3ccc6 in term 0.
I20260619 18:40:08.841576 13918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee53c95cd3354facb37b89c399d3ccc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" is_pre_election: true
I20260619 18:40:08.841889 13918 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee53c95cd3354facb37b89c399d3ccc6 in term 0.
I20260619 18:40:08.842659 13741 leader_election.cc:302] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f03a4bfc87b41e282a342b60f6a22d0, ee53c95cd3354facb37b89c399d3ccc6; no voters:
I20260619 18:40:08.843287 13974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:40:08.843380 13974 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:40:08.843420 13974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:08.844748 13974 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } }
I20260619 18:40:08.845122 13974 leader_election.cc:288] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [CANDIDATE]: Term 1 election: Requested vote from peers 66a53bf865d344728cad887eeb8b6920 (127.0.98.61:42001), 3f03a4bfc87b41e282a342b60f6a22d0 (127.0.98.60:45647)
I20260619 18:40:08.847579 13918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee53c95cd3354facb37b89c399d3ccc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f03a4bfc87b41e282a342b60f6a22d0"
I20260619 18:40:08.847749 13918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:08.849162 13918 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee53c95cd3354facb37b89c399d3ccc6 in term 1.
I20260619 18:40:08.849633 13841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee53c95cd3354facb37b89c399d3ccc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66a53bf865d344728cad887eeb8b6920"
I20260619 18:40:08.849792 13841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:08.851110 13841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee53c95cd3354facb37b89c399d3ccc6 in term 1.
I20260619 18:40:08.851576 13741 leader_election.cc:302] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66a53bf865d344728cad887eeb8b6920, ee53c95cd3354facb37b89c399d3ccc6; no voters:
I20260619 18:40:08.852003 13974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:08.852298 13974 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [term 1 LEADER]: Becoming Leader. State: Replica: ee53c95cd3354facb37b89c399d3ccc6, State: Running, Role: LEADER
I20260619 18:40:08.852856 13974 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } }
I20260619 18:40:08.857066 13974 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee53c95cd3354facb37b89c399d3ccc6. Latest consensus state: current_term: 1 leader_uuid: "ee53c95cd3354facb37b89c399d3ccc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } } }
I20260619 18:40:08.858145 13974 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:08.858568 14002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:40:08.859510 14002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:40:08.866623 13918 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0 [term 1 FOLLOWER]: Refusing update from remote peer ee53c95cd3354facb37b89c399d3ccc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:08.867609 14000 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:08.872006 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee53c95cd3354facb37b89c399d3ccc6. Latest consensus state: current_term: 1 leader_uuid: "ee53c95cd3354facb37b89c399d3ccc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } } }
I20260619 18:40:08.872144 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:40:08.873315 13841 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920 [term 1 FOLLOWER]: Refusing update from remote peer ee53c95cd3354facb37b89c399d3ccc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:08.874003 14000 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:08.877971 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee53c95cd3354facb37b89c399d3ccc6. Latest consensus state: current_term: 1 leader_uuid: "ee53c95cd3354facb37b89c399d3ccc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } } }
I20260619 18:40:08.878098 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:40:08.886621 14000 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee53c95cd3354facb37b89c399d3ccc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } } }
I20260619 18:40:08.886796 14000 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:08.887295 14003 mvcc.cc:205] Tried to move back new op lower bound from 7298639498709225472 to 7298639498674327552. Current Snapshot: MvccSnapshot[applied={T|T < 7298639498709225472}]
W20260619 18:40:08.888073 13897 tablet.cc:2369] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:40:08.888309 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee53c95cd3354facb37b89c399d3ccc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } } }
I20260619 18:40:08.888424 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:40:08.889230 14002 catalog_manager.cc:1347] Generated new cluster ID: dc83ee50bc114047a7300ab6cf68c06f
I20260619 18:40:08.889289 14002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:40:08.889623 14006 mvcc.cc:205] Tried to move back new op lower bound from 7298639498709225472 to 7298639498674327552. Current Snapshot: MvccSnapshot[applied={T|T < 7298639498709225472}]
I20260619 18:40:08.890244 13974 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ee53c95cd3354facb37b89c399d3ccc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } } }
I20260619 18:40:08.890347 13974 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6 [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:08.891712 14004 mvcc.cc:205] Tried to move back new op lower bound from 7298639498709225472 to 7298639498674327552. Current Snapshot: MvccSnapshot[applied={T|T < 7298639498709225472}]
I20260619 18:40:08.892040 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ee53c95cd3354facb37b89c399d3ccc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee53c95cd3354facb37b89c399d3ccc6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34003 } } peers { permanent_uuid: "66a53bf865d344728cad887eeb8b6920" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42001 } } peers { permanent_uuid: "3f03a4bfc87b41e282a342b60f6a22d0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45647 } } }
I20260619 18:40:08.892141 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:40:08.908375 14002 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:40:08.909773 14002 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:40:08.922674 14002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee53c95cd3354facb37b89c399d3ccc6: Generated new TSK 0
I20260619 18:40:08.923508 14002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:40:09.712361 13970 catalog_manager.cc:1259] Loaded cluster ID: dc83ee50bc114047a7300ab6cf68c06f
I20260619 18:40:09.712443 13970 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0: loading cluster ID for follower catalog manager: success
I20260619 18:40:09.714044 13970 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0: acquiring CA information for follower catalog manager: success
I20260619 18:40:09.715119 13970 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3f03a4bfc87b41e282a342b60f6a22d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:40:09.727823 13997 catalog_manager.cc:1259] Loaded cluster ID: dc83ee50bc114047a7300ab6cf68c06f
I20260619 18:40:09.727896 13997 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920: loading cluster ID for follower catalog manager: success
I20260619 18:40:09.729487 13997 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920: acquiring CA information for follower catalog manager: success
I20260619 18:40:09.730633 13997 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 66a53bf865d344728cad887eeb8b6920: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:40:09.990135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 13725
I20260619 18:40:09.995658 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 13800
I20260619 18:40:10.000648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 13875
2026-06-19T18:40:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3033 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:40:10 dist-test-slave-rxv6 krb5kdc[14017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:40:10 dist-test-slave-rxv6 krb5kdc[14017](info): set up 2 sockets
Jun 19 18:40:10 dist-test-slave-rxv6 krb5kdc[14017](info): commencing operation
krb5kdc: starting...
W20260619 18:40:10.957818 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:40:11 dist-test-slave-rxv6 krb5kdc[14017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:40:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:40:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:40:11.070180 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45503
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:11.177443 14030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:11.177713 14030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:11.177754 14030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:11.181770 14030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:40:11.181838 14030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:11.181856 14030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:11.181872 14030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:40:11.181886 14030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:40:11.187142 14030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:11.188169 14030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:11.189154 14030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:11.197436 14038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:11.197443 14036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:11.197635 14030 server_base.cc:1034] running on GCE node
W20260619 18:40:11.197685 14035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:11.198331 14030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:11.199412 14030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:11.200577 14030 hybrid_clock.cc:648] HybridClock initialized: now 1781894411200559 us; error 30 us; skew 500 ppm
Jun 19 18:40:11 dist-test-slave-rxv6 krb5kdc[14017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:11.203331 14030 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:40:11.204201 14030 webserver.cc:460] Webserver started at http://127.0.98.62:36861/ using document root <none> and password file <none>
I20260619 18:40:11.204723 14030 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:11.204779 14030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:11.204939 14030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:11.207065 14030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "ad337c3aeef8489a8190877a85dcf4ea"
format_stamp: "Formatted at 2026-06-19 18:40:11 on dist-test-slave-rxv6"
server_key: "b86488f0a517713546e0876084e77e79"
server_key_iv: "dac091bcb0a84ed0f26b239ab05c3fb1"
server_key_version: "encryptionkey@0"
I20260619 18:40:11.207715 14030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "ad337c3aeef8489a8190877a85dcf4ea"
format_stamp: "Formatted at 2026-06-19 18:40:11 on dist-test-slave-rxv6"
server_key: "b86488f0a517713546e0876084e77e79"
server_key_iv: "dac091bcb0a84ed0f26b239ab05c3fb1"
server_key_version: "encryptionkey@0"
I20260619 18:40:11.211436 14030 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:40:11.214027 14045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:11.215284 14030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:40:11.215399 14030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "ad337c3aeef8489a8190877a85dcf4ea"
format_stamp: "Formatted at 2026-06-19 18:40:11 on dist-test-slave-rxv6"
server_key: "b86488f0a517713546e0876084e77e79"
server_key_iv: "dac091bcb0a84ed0f26b239ab05c3fb1"
server_key_version: "encryptionkey@0"
I20260619 18:40:11.215500 14030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:11.228473 14030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:11.231400 14030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:11.231573 14030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:11.238436 14030 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45503
I20260619 18:40:11.238463 14097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45503 every 8 connection(s)
I20260619 18:40:11.239384 14030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:40:11.242259 14098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:11.245792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 14030
I20260619 18:40:11.245929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:40:11.246227 392 external_mini_cluster.cc:1428] Setting key 924ea2da8f3d5b1f6ccaad4aaecd5453
I20260619 18:40:11.248332 14098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea: Bootstrap starting.
I20260619 18:40:11.251044 14098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:11.251899 14098 log.cc:824] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:11.254041 14098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea: No bootstrap required, opened a new log
Jun 19 18:40:11 dist-test-slave-rxv6 krb5kdc[14017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:11.258329 14098 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad337c3aeef8489a8190877a85dcf4ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260619 18:40:11.258598 14098 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:11.258637 14098 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad337c3aeef8489a8190877a85dcf4ea, State: Initialized, Role: FOLLOWER
I20260619 18:40:11.259110 14098 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad337c3aeef8489a8190877a85dcf4ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260619 18:40:11.259245 14098 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:40:11.259310 14098 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:40:11.259406 14098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:11.260324 14098 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad337c3aeef8489a8190877a85dcf4ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260619 18:40:11.260663 14098 leader_election.cc:302] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad337c3aeef8489a8190877a85dcf4ea; no voters:
I20260619 18:40:11.260951 14098 leader_election.cc:288] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:40:11.261051 14103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:11.261233 14103 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [term 1 LEADER]: Becoming Leader. State: Replica: ad337c3aeef8489a8190877a85dcf4ea, State: Running, Role: LEADER
I20260619 18:40:11.261421 14101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:11.247564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58306 (local address 127.0.98.62:45503)
0619 18:40:11.247992 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:11.248011 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:11.248053 (+ 42us) server_negotiation.cc:407] Connection header received
0619 18:40:11.248868 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:11.248891 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:11.249295 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:11.249674 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:11.250345 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:11.251828 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:11.252794 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:11.253469 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:11.253634 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:11.256222 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:11.256229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:11.256246 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:11.256278 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:11.258273 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:11.258762 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:11.258767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:11.258772 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:11.258891 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:11.259213 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:11.259215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:11.259217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:11.259713 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:11.259912 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:11.260192 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 18:40:11.260517 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":130,"threads_started":1}
I20260619 18:40:11.261780 14103 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad337c3aeef8489a8190877a85dcf4ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260619 18:40:11.262316 14098 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:40:11.263392 14104 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad337c3aeef8489a8190877a85dcf4ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad337c3aeef8489a8190877a85dcf4ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } } }
I20260619 18:40:11.263511 14104 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:11.263825 14105 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad337c3aeef8489a8190877a85dcf4ea. Latest consensus state: current_term: 1 leader_uuid: "ad337c3aeef8489a8190877a85dcf4ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad337c3aeef8489a8190877a85dcf4ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } } }
I20260619 18:40:11.263895 14112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:40:11.263907 14105 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:11.265110 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:40:11.267000 14112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:40:11.272378 14112 catalog_manager.cc:1347] Generated new cluster ID: 4f58c2ddd7534effbc47e61fb90bcf43
I20260619 18:40:11.272464 14112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:40:11.285000 14112 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:40:11.286048 14112 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:40:11.298055 14112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad337c3aeef8489a8190877a85dcf4ea: Generated new TSK 0
I20260619 18:40:11.298780 14112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:40:11.321101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 14030
2026-06-19T18:40:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1319 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:40:11 dist-test-slave-rxv6 krb5kdc[14125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:40:11 dist-test-slave-rxv6 krb5kdc[14125](info): set up 2 sockets
Jun 19 18:40:11 dist-test-slave-rxv6 krb5kdc[14125](info): commencing operation
krb5kdc: starting...
W20260619 18:40:12.277827 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:40:12 dist-test-slave-rxv6 krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:40:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:40:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:40:12.390828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38743
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:12.496270 14138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:12.496526 14138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:12.496564 14138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:12.500458 14138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:40:12.500514 14138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:12.500532 14138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:12.500547 14138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:40:12.500560 14138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:40:12.505635 14138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:12.506616 14138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:12.507527 14138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:12.515400 14144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:12.515400 14146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:12.515564 14143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:12.515686 14138 server_base.cc:1034] running on GCE node
I20260619 18:40:12.516204 14138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:12.517251 14138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:12.518409 14138 hybrid_clock.cc:648] HybridClock initialized: now 1781894412518391 us; error 25 us; skew 500 ppm
Jun 19 18:40:12 dist-test-slave-rxv6 krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:12.520998 14138 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:40:12.521814 14138 webserver.cc:460] Webserver started at http://127.0.98.62:41339/ using document root <none> and password file <none>
I20260619 18:40:12.522341 14138 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:12.522398 14138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:12.522552 14138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:12.524442 14138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "273f071c29b34623a4b14211296b834f"
format_stamp: "Formatted at 2026-06-19 18:40:12 on dist-test-slave-rxv6"
server_key: "9c1f08011dae9fcf57047f4ed3b9b395"
server_key_iv: "da188f8d7234f478fe7e09fbf816bdf8"
server_key_version: "encryptionkey@0"
I20260619 18:40:12.524921 14138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "273f071c29b34623a4b14211296b834f"
format_stamp: "Formatted at 2026-06-19 18:40:12 on dist-test-slave-rxv6"
server_key: "9c1f08011dae9fcf57047f4ed3b9b395"
server_key_iv: "da188f8d7234f478fe7e09fbf816bdf8"
server_key_version: "encryptionkey@0"
I20260619 18:40:12.528409 14138 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:40:12.530683 14153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:12.531896 14138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:40:12.532011 14138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "273f071c29b34623a4b14211296b834f"
format_stamp: "Formatted at 2026-06-19 18:40:12 on dist-test-slave-rxv6"
server_key: "9c1f08011dae9fcf57047f4ed3b9b395"
server_key_iv: "da188f8d7234f478fe7e09fbf816bdf8"
server_key_version: "encryptionkey@0"
I20260619 18:40:12.532110 14138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:12.558233 14138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:12.561288 14138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:12.561467 14138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:12.568086 14138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38743
I20260619 18:40:12.568338 14205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38743 every 8 connection(s)
I20260619 18:40:12.569011 14138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:40:12.571733 14206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:12.576463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 14138
I20260619 18:40:12.576565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:40:12.576808 392 external_mini_cluster.cc:1428] Setting key b635222b3784b5e57d2e5564f99399bf
I20260619 18:40:12.577086 14206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f: Bootstrap starting.
I20260619 18:40:12.579735 14206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:12.580587 14206 log.cc:824] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:12.582489 14206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f: No bootstrap required, opened a new log
Jun 19 18:40:12 dist-test-slave-rxv6 krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:12.585834 14206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } }
I20260619 18:40:12.586086 14206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:12.586154 14206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273f071c29b34623a4b14211296b834f, State: Initialized, Role: FOLLOWER
I20260619 18:40:12.586591 14206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } }
I20260619 18:40:12.586717 14206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:40:12.586783 14206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:40:12.586881 14206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:12.587700 14206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } }
I20260619 18:40:12.588052 14206 leader_election.cc:302] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 273f071c29b34623a4b14211296b834f; no voters:
I20260619 18:40:12.588411 14206 leader_election.cc:288] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:40:12.588524 14211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:12.588752 14211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 1 LEADER]: Becoming Leader. State: Replica: 273f071c29b34623a4b14211296b834f, State: Running, Role: LEADER
I20260619 18:40:12.589097 14211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } }
I20260619 18:40:12.589723 14206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:40:12.590828 14212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "273f071c29b34623a4b14211296b834f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } }
I20260619 18:40:12.590938 14212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:12.591061 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:12.578228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52382 (local address 127.0.98.62:38743)
0619 18:40:12.578635 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:12.578654 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:12.578695 (+ 41us) server_negotiation.cc:407] Connection header received
0619 18:40:12.579484 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:12.579508 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:12.579949 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:12.580310 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:12.580990 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:12.582067 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:12.582930 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:12.583393 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:12.583543 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:12.585782 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:12.585787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:12.585800 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:12.585827 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:12.587510 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:12.587911 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:12.587918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:12.587923 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:12.588065 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:12.588305 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:12.588308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:12.588309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:12.588830 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:12.589591 (+ 761us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:12.589736 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 18:40:12.590071 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":121,"threads_started":1}
I20260619 18:40:12.591197 14213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 273f071c29b34623a4b14211296b834f. Latest consensus state: current_term: 1 leader_uuid: "273f071c29b34623a4b14211296b834f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } }
I20260619 18:40:12.591274 14213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:12.591336 14220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:40:12.594377 14220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:40:12.599457 14220 catalog_manager.cc:1347] Generated new cluster ID: 6114c5649a084e07b9e367ef0981fa2a
I20260619 18:40:12.599524 14220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:40:12.620584 14220 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:40:12.621578 14220 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:40:12.626415 14220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f: Generated new TSK 0
I20260619 18:40:12.627079 14220 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:40:12.657392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38743
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:12.769276 14232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:12.769549 14232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:12.769593 14232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:12.773622 14232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:12.773692 14232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:12.773752 14232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:40:12.779057 14232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:12.780112 14232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:12.781083 14232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:12.790582 14240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:12.790589 14238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:12.790776 14232 server_base.cc:1034] running on GCE node
W20260619 18:40:12.790582 14237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:12.791371 14232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:12.792004 14232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:12.793171 14232 hybrid_clock.cc:648] HybridClock initialized: now 1781894412793146 us; error 37 us; skew 500 ppm
Jun 19 18:40:12 dist-test-slave-rxv6 krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:12.795925 14232 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:40:12.796869 14232 webserver.cc:460] Webserver started at http://127.0.98.1:37291/ using document root <none> and password file <none>
I20260619 18:40:12.797415 14232 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:12.797472 14232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:12.797658 14232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:12.799711 14232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "7fbe0fb5b8c54df5a614d2474e3b9625"
format_stamp: "Formatted at 2026-06-19 18:40:12 on dist-test-slave-rxv6"
server_key: "0b9be3c77854a8afe75872ead2d1e1a5"
server_key_iv: "a2e4cf771129f9a389e3cb6eeb203dd4"
server_key_version: "encryptionkey@0"
I20260619 18:40:12.800228 14232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7fbe0fb5b8c54df5a614d2474e3b9625"
format_stamp: "Formatted at 2026-06-19 18:40:12 on dist-test-slave-rxv6"
server_key: "0b9be3c77854a8afe75872ead2d1e1a5"
server_key_iv: "a2e4cf771129f9a389e3cb6eeb203dd4"
server_key_version: "encryptionkey@0"
I20260619 18:40:12.803947 14232 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:40:12.806367 14247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:12.807612 14232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:40:12.807726 14232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "7fbe0fb5b8c54df5a614d2474e3b9625"
format_stamp: "Formatted at 2026-06-19 18:40:12 on dist-test-slave-rxv6"
server_key: "0b9be3c77854a8afe75872ead2d1e1a5"
server_key_iv: "a2e4cf771129f9a389e3cb6eeb203dd4"
server_key_version: "encryptionkey@0"
I20260619 18:40:12.807840 14232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:12.828706 14232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:12.831710 14232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:12.831986 14232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:12.832700 14232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:12.833599 14232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:12.833648 14232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:12.833705 14232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:12.833726 14232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:12.842595 14232 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42283
I20260619 18:40:12.842665 14360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42283 every 8 connection(s)
I20260619 18:40:12.843478 14232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:40:12.843853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 14232
I20260619 18:40:12.843990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:40:12.844288 392 external_mini_cluster.cc:1428] Setting key 21b1c9ed527e8285cd7258c0f8fbcb8f
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Jun 19 18:40:12 dist-test-slave-rxv6 krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 18:40:12.857784 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:12.845201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38561 (local address 127.0.98.62:38743)
0619 18:40:12.845348 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:12.845352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:12.846332 (+ 980us) server_negotiation.cc:407] Connection header received
0619 18:40:12.847569 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:12.847573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:12.847626 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:12.847716 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:12.849134 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:12.849705 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:12.851381 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:12.851608 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:12.851688 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:12.854262 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:12.854265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:12.854268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:12.854302 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:12.855761 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:12.856261 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:12.856264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:12.856266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:12.856352 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:12.856667 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:12.856670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:12.856672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:12.856856 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:12.856939 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:12.857549 (+ 610us) server_negotiation.cc:299] Negotiation successful
0619 18:40:12.857656 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:40:12.858843 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:12.845655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:38561)
0619 18:40:12.846164 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:12.846227 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:12.847373 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:12.847856 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:12.847863 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:12.848349 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:12.848980 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:12.849000 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:12.849838 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:12.849842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:12.851258 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:12.851270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:12.851754 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:12.851758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:12.851949 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:12.852535 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:12.852555 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:12.854109 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:12.855861 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:12.855867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:12.855882 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:12.856160 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:12.856431 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:12.856434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:12.856436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:12.856575 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:12.856953 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:12.856959 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:12.857272 (+ 313us) client_negotiation.cc:769] Sending connection context
0619 18:40:12.857469 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 18:40:12.857797 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":100,"threads_started":1}
I20260619 18:40:12.859982 14361 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38743
I20260619 18:40:12.860205 14361 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:12.860661 14361 heartbeater.cc:511] Master 127.0.98.62:38743 requested a full tablet report, sending...
I20260619 18:40:12.862187 14170 ts_manager.cc:194] Registered new tserver with Master: 7fbe0fb5b8c54df5a614d2474e3b9625 (127.0.98.1:42283)
I20260619 18:40:12.863327 14170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:40:12.870428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38743
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:12.977173 14366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:12.977424 14366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:12.977465 14366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:12.981359 14366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:12.981418 14366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:12.981478 14366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:40:12.986625 14366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:12.987617 14366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:12.988570 14366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:12.997776 14371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:12.998015 14366 server_base.cc:1034] running on GCE node
W20260619 18:40:12.997965 14374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:12.997952 14372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:12.998560 14366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:12.999159 14366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:13.000319 14366 hybrid_clock.cc:648] HybridClock initialized: now 1781894413000300 us; error 29 us; skew 500 ppm
Jun 19 18:40:13 dist-test-slave-rxv6 krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:13.002915 14366 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:40:13.003691 14366 webserver.cc:460] Webserver started at http://127.0.98.2:37253/ using document root <none> and password file <none>
I20260619 18:40:13.004180 14366 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:13.004232 14366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:13.004384 14366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:13.006356 14366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "a8db4a68a7e648648e189299751e2263"
format_stamp: "Formatted at 2026-06-19 18:40:13 on dist-test-slave-rxv6"
server_key: "0fd2acb7d03eee65d6d8f41478421e07"
server_key_iv: "e240b15ed1a85d8c5696b8f65616001b"
server_key_version: "encryptionkey@0"
I20260619 18:40:13.006853 14366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a8db4a68a7e648648e189299751e2263"
format_stamp: "Formatted at 2026-06-19 18:40:13 on dist-test-slave-rxv6"
server_key: "0fd2acb7d03eee65d6d8f41478421e07"
server_key_iv: "e240b15ed1a85d8c5696b8f65616001b"
server_key_version: "encryptionkey@0"
I20260619 18:40:13.010377 14366 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:40:13.012813 14381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:13.014185 14366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:40:13.014297 14366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "a8db4a68a7e648648e189299751e2263"
format_stamp: "Formatted at 2026-06-19 18:40:13 on dist-test-slave-rxv6"
server_key: "0fd2acb7d03eee65d6d8f41478421e07"
server_key_iv: "e240b15ed1a85d8c5696b8f65616001b"
server_key_version: "encryptionkey@0"
I20260619 18:40:13.014405 14366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:13.033659 14366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:13.038750 14366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:13.038950 14366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:13.039563 14366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:13.040414 14366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:13.040462 14366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:13.040517 14366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:13.040546 14366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:13.048806 14366 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45675
I20260619 18:40:13.048887 14494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45675 every 8 connection(s)
I20260619 18:40:13.049767 14366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:40:13.057224 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 14366
I20260619 18:40:13.057353 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:40:13.057648 392 external_mini_cluster.cc:1428] Setting key 25f8869dfa14c44ffcf2de3e5268342d
Jun 19 18:40:13 dist-test-slave-rxv6 krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:13.064837 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.051906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53953 (local address 127.0.98.62:38743)
0619 18:40:13.052094 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:13.052098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:13.052931 (+ 833us) server_negotiation.cc:407] Connection header received
0619 18:40:13.054294 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:13.054298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:13.054358 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:13.054436 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:13.056151 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.056707 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.058326 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.058556 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.058647 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:13.061338 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:13.061341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:13.061345 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:13.061374 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:13.062860 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:13.063361 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:13.063364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:13.063366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:13.063444 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:13.063763 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:13.063767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:13.063768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:13.063954 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:13.064037 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:13.064542 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 18:40:13.064665 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:40:13.065663 14497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.052260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:53953)
0619 18:40:13.052757 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:13.052801 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:13.054074 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:13.054626 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:13.054639 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:13.055344 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:13.055923 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:13.055938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.056838 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.056841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:13.058202 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:13.058212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.058699 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.058703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:13.058883 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:13.059485 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:13.059507 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:13.061145 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:13.062973 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:13.062979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:13.062997 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:13.063266 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:13.063526 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:13.063529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:13.063531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:13.063674 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:13.064048 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:13.064055 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:13.064315 (+ 260us) client_negotiation.cc:769] Sending connection context
0619 18:40:13.064482 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 18:40:13.064810 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 18:40:13.066943 14495 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38743
I20260619 18:40:13.067175 14495 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:13.067688 14495 heartbeater.cc:511] Master 127.0.98.62:38743 requested a full tablet report, sending...
I20260619 18:40:13.068804 14170 ts_manager.cc:194] Registered new tserver with Master: a8db4a68a7e648648e189299751e2263 (127.0.98.2:45675)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 18:40:13.069602 14170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:40:13.084769 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38743
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:13.193744 14500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:13.194047 14500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:13.194096 14500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:13.198100 14500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:13.198158 14500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:13.198217 14500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:40:13.203459 14500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:13.204493 14500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:13.205461 14500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:13.214975 14506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:13.214986 14508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:13.214951 14505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:13.214967 14500 server_base.cc:1034] running on GCE node
I20260619 18:40:13.215502 14500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:13.216143 14500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:13.217319 14500 hybrid_clock.cc:648] HybridClock initialized: now 1781894413217294 us; error 36 us; skew 500 ppm
Jun 19 18:40:13 dist-test-slave-rxv6 krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:13.223073 14500 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:40:13.223914 14500 webserver.cc:460] Webserver started at http://127.0.98.3:44413/ using document root <none> and password file <none>
I20260619 18:40:13.224421 14500 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:13.224475 14500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:13.224635 14500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:13.226626 14500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "28bd1f116dca4597bdd411bc83987f53"
format_stamp: "Formatted at 2026-06-19 18:40:13 on dist-test-slave-rxv6"
server_key: "310c9e7772acb48a8d8f2fd85f853519"
server_key_iv: "650991851c9c50b0ed2e4d938584383b"
server_key_version: "encryptionkey@0"
I20260619 18:40:13.227115 14500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "28bd1f116dca4597bdd411bc83987f53"
format_stamp: "Formatted at 2026-06-19 18:40:13 on dist-test-slave-rxv6"
server_key: "310c9e7772acb48a8d8f2fd85f853519"
server_key_iv: "650991851c9c50b0ed2e4d938584383b"
server_key_version: "encryptionkey@0"
I20260619 18:40:13.230695 14500 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:40:13.232978 14515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:13.234303 14500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:40:13.234411 14500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "28bd1f116dca4597bdd411bc83987f53"
format_stamp: "Formatted at 2026-06-19 18:40:13 on dist-test-slave-rxv6"
server_key: "310c9e7772acb48a8d8f2fd85f853519"
server_key_iv: "650991851c9c50b0ed2e4d938584383b"
server_key_version: "encryptionkey@0"
I20260619 18:40:13.234498 14500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:13.247706 14500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:13.250737 14500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:13.250907 14500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:13.251475 14500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:13.252290 14500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:13.252336 14500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:13.252385 14500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:13.252414 14500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:13.260982 14500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46683
I20260619 18:40:13.261013 14628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46683 every 8 connection(s)
I20260619 18:40:13.261890 14500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:40:13.270789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 14500
I20260619 18:40:13.270912 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:40:13.271211 392 external_mini_cluster.cc:1428] Setting key 1b26b45d58869ea0a7a505f275af1f33
Jun 19 18:40:13 dist-test-slave-rxv6 krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:13.275745 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.263822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37411 (local address 127.0.98.62:38743)
0619 18:40:13.264003 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:13.264007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:13.264769 (+ 762us) server_negotiation.cc:407] Connection header received
0619 18:40:13.265991 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:13.265995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:13.266054 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:13.266134 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:13.267375 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.267925 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.269063 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.269283 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.269374 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:13.272079 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:13.272083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:13.272085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:13.272114 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:13.273611 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:13.274209 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:13.274212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:13.274214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:13.274303 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:13.274672 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:13.274675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:13.274676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:13.274876 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:13.274952 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:13.275512 (+ 560us) server_negotiation.cc:299] Negotiation successful
0619 18:40:13.275625 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 18:40:13.276566 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.264167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:37411)
0619 18:40:13.264611 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:13.264653 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:13.265773 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:13.266355 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:13.266362 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:13.266736 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:13.267225 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:13.267239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.268053 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.268055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:13.268945 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:13.268955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.269424 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.269428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:13.269596 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:13.270202 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:13.270219 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:13.271878 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:13.273711 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:13.273716 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:13.273730 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:13.274065 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:13.274424 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:13.274427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:13.274428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:13.274566 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:13.274987 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:13.274993 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:13.275249 (+ 256us) client_negotiation.cc:769] Sending connection context
0619 18:40:13.275432 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 18:40:13.275711 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":108,"threads_started":1}
I20260619 18:40:13.277678 14629 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38743
I20260619 18:40:13.277972 14629 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:13.278482 14629 heartbeater.cc:511] Master 127.0.98.62:38743 requested a full tablet report, sending...
I20260619 18:40:13.279512 14170 ts_manager.cc:194] Registered new tserver with Master: 28bd1f116dca4597bdd411bc83987f53 (127.0.98.3:46683)
I20260619 18:40:13.280076 14170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37411
I20260619 18:40:13.285245 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:40:13.294569 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.287237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.0.98.62:38743)
0619 18:40:13.287403 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:13.287407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:13.287501 (+ 94us) server_negotiation.cc:407] Connection header received
0619 18:40:13.287601 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:13.287603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:13.287643 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:13.287712 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:13.288434 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.288962 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.289800 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.290117 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.290208 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:13.291540 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:13.291543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:13.291545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:13.291569 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:13.292875 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:13.293278 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:13.293282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:13.293283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:13.293411 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:13.293674 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:13.293677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:13.293678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:13.293942 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:13.294062 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:13.294317 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 18:40:13.294448 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:40:13.297524 14170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:40:13.300544 14170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:40:13.315856 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.309040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42283 (local address 127.0.0.1:34782)
0619 18:40:13.309438 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:13.309484 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:13.309604 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:13.310844 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:13.310850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:13.310876 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:13.311040 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:13.311047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.312647 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.312650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:13.314158 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:13.314168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.315269 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.315273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:13.315384 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:13.315426 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 18:40:13.315540 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 18:40:13.315673 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":76,"spinlock_wait_cycles":10496,"thread_start_us":74,"threads_started":1}
I20260619 18:40:13.315887 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.309188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.0.98.1:42283)
0619 18:40:13.310304 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:13.310327 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:13.310351 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:40:13.310430 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:13.310435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:13.310647 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:13.311173 (+ 526us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:13.311292 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.312449 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.314277 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.315146 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.315260 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:13.315411 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:13.315582 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 18:40:13.315679 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":74,"threads_started":1}
I20260619 18:40:13.316334 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.309986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33084 (local address 127.0.98.3:46683)
0619 18:40:13.310366 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:13.310377 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:13.311022 (+ 645us) server_negotiation.cc:407] Connection header received
0619 18:40:13.311090 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:13.311095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:13.311239 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:13.311405 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:13.311786 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.313077 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.314890 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.315541 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.315633 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:13.315759 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:13.315872 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 18:40:13.315968 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":89,"threads_started":1}
I20260619 18:40:13.316560 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.309119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45675 (local address 127.0.0.1:34916)
0619 18:40:13.309681 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:13.309695 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:13.309783 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:13.311442 (+ 1659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:13.311445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:13.311462 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:13.311543 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:13.311548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.312512 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.312515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:13.314218 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:13.314229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.316167 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.316171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:13.316262 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:13.316285 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:40:13.316355 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 18:40:13.316437 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":248,"thread_start_us":120,"threads_started":1}
I20260619 18:40:13.316979 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.310453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34916 (local address 127.0.98.2:45675)
0619 18:40:13.310945 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:13.310955 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:13.310979 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:40:13.311054 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:13.311059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:13.311318 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:13.311448 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:13.311638 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.312393 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.314421 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.315159 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.315260 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:13.315411 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:13.316741 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0619 18:40:13.316844 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":92,"threads_started":1}
I20260619 18:40:13.318089 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.309731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46683 (local address 127.0.0.1:33084)
0619 18:40:13.310826 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:13.310840 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:13.310932 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:13.311565 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:13.311568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:13.311585 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:13.311672 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:13.311679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.313878 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.313881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:13.314770 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:13.314779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.315644 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.315648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:13.315730 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:13.315753 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:40:13.315836 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 18:40:13.315917 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"mutex_wait_us":11,"thread_start_us":87,"threads_started":1}
I20260619 18:40:13.319119 14295 tablet_service.cc:1467] Processing CreateTablet for tablet 231a0fdd0df24e68971cada47fd260ba (DEFAULT_TABLE table=test-table [id=405f3bbd265742ba9c458363a8988380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:13.319556 14429 tablet_service.cc:1467] Processing CreateTablet for tablet 231a0fdd0df24e68971cada47fd260ba (DEFAULT_TABLE table=test-table [id=405f3bbd265742ba9c458363a8988380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:13.320117 14295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 231a0fdd0df24e68971cada47fd260ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:13.320508 14429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 231a0fdd0df24e68971cada47fd260ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:13.326098 14646 tablet_bootstrap.cc:492] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625: Bootstrap starting.
I20260619 18:40:13.326241 14647 tablet_bootstrap.cc:492] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: Bootstrap starting.
I20260619 18:40:13.328272 14646 tablet_bootstrap.cc:654] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:13.328716 14647 tablet_bootstrap.cc:654] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:13.329559 14646 log.cc:824] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:13.329129 14563 tablet_service.cc:1467] Processing CreateTablet for tablet 231a0fdd0df24e68971cada47fd260ba (DEFAULT_TABLE table=test-table [id=405f3bbd265742ba9c458363a8988380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:13.330056 14647 log.cc:824] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:13.330327 14563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 231a0fdd0df24e68971cada47fd260ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:13.331502 14646 tablet_bootstrap.cc:492] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625: No bootstrap required, opened a new log
I20260619 18:40:13.331705 14646 ts_tablet_manager.cc:1397] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260619 18:40:13.331913 14647 tablet_bootstrap.cc:492] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: No bootstrap required, opened a new log
I20260619 18:40:13.332086 14647 ts_tablet_manager.cc:1397] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 18:40:13.335683 14647 raft_consensus.cc:348] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } }
I20260619 18:40:13.335683 14646 raft_consensus.cc:348] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } }
I20260619 18:40:13.335958 14646 raft_consensus.cc:374] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:13.335994 14647 raft_consensus.cc:374] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:13.336033 14646 raft_consensus.cc:729] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbe0fb5b8c54df5a614d2474e3b9625, State: Initialized, Role: FOLLOWER
I20260619 18:40:13.336068 14647 raft_consensus.cc:729] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8db4a68a7e648648e189299751e2263, State: Initialized, Role: FOLLOWER
I20260619 18:40:13.336513 14646 consensus_queue.cc:260] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } }
I20260619 18:40:13.336513 14647 consensus_queue.cc:260] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } }
I20260619 18:40:13.337165 14495 heartbeater.cc:503] Master 127.0.98.62:38743 was elected leader, sending a full tablet report...
I20260619 18:40:13.337502 14646 ts_tablet_manager.cc:1428] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260619 18:40:13.337502 14647 ts_tablet_manager.cc:1428] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260619 18:40:13.337837 14361 heartbeater.cc:503] Master 127.0.98.62:38743 was elected leader, sending a full tablet report...
I20260619 18:40:13.338176 14650 tablet_bootstrap.cc:492] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53: Bootstrap starting.
I20260619 18:40:13.340576 14650 tablet_bootstrap.cc:654] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:13.341732 14650 log.cc:824] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:13.343501 14650 tablet_bootstrap.cc:492] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53: No bootstrap required, opened a new log
I20260619 18:40:13.343693 14650 ts_tablet_manager.cc:1397] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
W20260619 18:40:13.344624 14362 tablet.cc:2369] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:40:13.347007 14650 raft_consensus.cc:348] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } }
I20260619 18:40:13.347229 14650 raft_consensus.cc:374] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:13.347261 14650 raft_consensus.cc:729] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28bd1f116dca4597bdd411bc83987f53, State: Initialized, Role: FOLLOWER
I20260619 18:40:13.347606 14650 consensus_queue.cc:260] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } }
I20260619 18:40:13.348165 14629 heartbeater.cc:503] Master 127.0.98.62:38743 was elected leader, sending a full tablet report...
I20260619 18:40:13.348404 14650 ts_tablet_manager.cc:1428] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 18:40:13.512909 14630 tablet.cc:2369] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:40:13.551151 14496 tablet.cc:2369] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:40:13.590365 14651 raft_consensus.cc:482] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:13.590639 14651 raft_consensus.cc:504] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } }
I20260619 18:40:13.592187 14651 leader_election.cc:288] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7fbe0fb5b8c54df5a614d2474e3b9625 (127.0.98.1:42283), 28bd1f116dca4597bdd411bc83987f53 (127.0.98.3:46683)
I20260619 18:40:13.596585 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.592619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49723 (local address 127.0.98.3:46683)
0619 18:40:13.592767 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:13.592773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:13.592987 (+ 214us) server_negotiation.cc:407] Connection header received
0619 18:40:13.593112 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:13.593116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:13.593184 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:13.593253 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:13.593547 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.594093 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.595443 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.596076 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.596171 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:13.596229 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:13.596393 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 18:40:13.596470 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:40:13.596583 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.592558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41635 (local address 127.0.98.1:42283)
0619 18:40:13.592708 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:13.592713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:13.592915 (+ 202us) server_negotiation.cc:407] Connection header received
0619 18:40:13.593088 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:13.593091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:13.593155 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:13.593225 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:13.593528 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.594101 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.595323 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.595938 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.596050 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:13.596128 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:13.596357 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 18:40:13.596441 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:40:13.596628 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.592468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46683 (local address 127.0.98.2:49723)
0619 18:40:13.592885 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:13.592897 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:13.593010 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:13.593280 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:13.593283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:13.593325 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:13.593449 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:13.593454 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.594207 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.594211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:13.595319 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:13.595329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.596187 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.596191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:13.596286 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:13.596305 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 18:40:13.596374 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 18:40:13.596435 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":85,"spinlock_wait_cycles":768,"thread_start_us":84,"threads_started":1}
I20260619 18:40:13.596671 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.592468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42283 (local address 127.0.98.2:41635)
0619 18:40:13.592823 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:13.592835 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:13.592963 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:13.593245 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:13.593248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:13.593264 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:13.593430 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:13.593436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.594210 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.594213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:13.595210 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:13.595219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.596100 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.596103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:13.596207 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:13.596238 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:40:13.596321 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 18:40:13.596417 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":54,"spinlock_wait_cycles":31360,"thread_start_us":87,"threads_started":1}
I20260619 18:40:13.597793 14583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231a0fdd0df24e68971cada47fd260ba" candidate_uuid: "a8db4a68a7e648648e189299751e2263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28bd1f116dca4597bdd411bc83987f53" is_pre_election: true
I20260619 18:40:13.597795 14315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231a0fdd0df24e68971cada47fd260ba" candidate_uuid: "a8db4a68a7e648648e189299751e2263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" is_pre_election: true
I20260619 18:40:13.598224 14315 raft_consensus.cc:2454] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8db4a68a7e648648e189299751e2263 in term 0.
I20260619 18:40:13.598224 14583 raft_consensus.cc:2454] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8db4a68a7e648648e189299751e2263 in term 0.
I20260619 18:40:13.598776 14385 leader_election.cc:302] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28bd1f116dca4597bdd411bc83987f53, a8db4a68a7e648648e189299751e2263; no voters:
I20260619 18:40:13.599135 14651 raft_consensus.cc:2781] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:40:13.599232 14651 raft_consensus.cc:482] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:40:13.599267 14651 raft_consensus.cc:3037] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:13.600627 14651 raft_consensus.cc:504] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } }
I20260619 18:40:13.601001 14651 leader_election.cc:288] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [CANDIDATE]: Term 1 election: Requested vote from peers 7fbe0fb5b8c54df5a614d2474e3b9625 (127.0.98.1:42283), 28bd1f116dca4597bdd411bc83987f53 (127.0.98.3:46683)
I20260619 18:40:13.601424 14315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231a0fdd0df24e68971cada47fd260ba" candidate_uuid: "a8db4a68a7e648648e189299751e2263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625"
I20260619 18:40:13.601425 14583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231a0fdd0df24e68971cada47fd260ba" candidate_uuid: "a8db4a68a7e648648e189299751e2263" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28bd1f116dca4597bdd411bc83987f53"
I20260619 18:40:13.601564 14315 raft_consensus.cc:3037] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:13.601564 14583 raft_consensus.cc:3037] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:13.603051 14315 raft_consensus.cc:2454] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8db4a68a7e648648e189299751e2263 in term 1.
I20260619 18:40:13.603046 14583 raft_consensus.cc:2454] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8db4a68a7e648648e189299751e2263 in term 1.
I20260619 18:40:13.603375 14384 leader_election.cc:302] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28bd1f116dca4597bdd411bc83987f53, 7fbe0fb5b8c54df5a614d2474e3b9625, a8db4a68a7e648648e189299751e2263; no voters:
I20260619 18:40:13.603559 14651 raft_consensus.cc:2781] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:13.603806 14651 raft_consensus.cc:686] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [term 1 LEADER]: Becoming Leader. State: Replica: a8db4a68a7e648648e189299751e2263, State: Running, Role: LEADER
I20260619 18:40:13.604238 14651 consensus_queue.cc:237] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } }
I20260619 18:40:13.607919 14170 catalog_manager.cc:5495] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8db4a68a7e648648e189299751e2263 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a8db4a68a7e648648e189299751e2263" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } health_report { overall_health: UNKNOWN } } }
I20260619 18:40:13.688961 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.684982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.0.98.2:45675)
0619 18:40:13.685142 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:13.685148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:13.685163 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:40:13.685263 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:13.685265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:13.685331 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:13.685444 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:13.685778 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.686397 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.687302 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:13.687526 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:13.687620 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:13.687945 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:13.688504 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:13.688621 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:13.688699 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 18:40:13.688773 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:40:13.697155 14315 raft_consensus.cc:1261] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Refusing update from remote peer a8db4a68a7e648648e189299751e2263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:13.697153 14583 raft_consensus.cc:1261] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53 [term 1 FOLLOWER]: Refusing update from remote peer a8db4a68a7e648648e189299751e2263: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:13.698226 14651 consensus_queue.cc:1035] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:13.698592 14657 consensus_queue.cc:1035] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:13.706918 14661 mvcc.cc:205] Tried to move back new op lower bound from 7298639518492569600 to 7298639518125862912. Current Snapshot: MvccSnapshot[applied={T|T < 7298639518492569600}]
I20260619 18:40:13.717286 14170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:13.717501 14170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:13.720518 14170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f: Sending DeleteTablet for 3 replicas of tablet 231a0fdd0df24e68971cada47fd260ba
I20260619 18:40:13.721530 14295 tablet_service.cc:1514] Processing DeleteTablet for tablet 231a0fdd0df24e68971cada47fd260ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34782
I20260619 18:40:13.721711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 14232
I20260619 18:40:13.721813 14429 tablet_service.cc:1514] Processing DeleteTablet for tablet 231a0fdd0df24e68971cada47fd260ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34916
I20260619 18:40:13.722431 14563 tablet_service.cc:1514] Processing DeleteTablet for tablet 231a0fdd0df24e68971cada47fd260ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33084
I20260619 18:40:13.723069 14670 tablet_replica.cc:330] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53: stopping tablet replica
I20260619 18:40:13.723503 14670 raft_consensus.cc:2229] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:13.723943 14670 raft_consensus.cc:2258] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 18:40:13.728781 14156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42283 (error 108)
I20260619 18:40:13.729476 14669 tablet_replica.cc:330] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: stopping tablet replica
I20260619 18:40:13.729755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 14366
I20260619 18:40:13.730875 14670 ts_tablet_manager.cc:1905] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:13.732486 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.732099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42283 (local address 127.0.0.1:34794)
0619 18:40:13.732275 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:13.732366 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 18:40:13.732700 14156 catalog_manager.cc:4584] TS 7fbe0fb5b8c54df5a614d2474e3b9625 (127.0.98.1:42283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 231a0fdd0df24e68971cada47fd260ba: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42283: connect: Connection refused (error 111)
I20260619 18:40:13.733870 14670 ts_tablet_manager.cc:1918] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:40:13.733999 14670 log.cc:1196] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/231a0fdd0df24e68971cada47fd260ba
I20260619 18:40:13.734282 14670 ts_tablet_manager.cc:1939] T 231a0fdd0df24e68971cada47fd260ba P 28bd1f116dca4597bdd411bc83987f53: Deleting consensus metadata
I20260619 18:40:13.735739 14157 catalog_manager.cc:4841] TS 28bd1f116dca4597bdd411bc83987f53 (127.0.98.3:46683): tablet 231a0fdd0df24e68971cada47fd260ba (table test-table [id=405f3bbd265742ba9c458363a8988380]) successfully deleted
I20260619 18:40:13.736821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 14500
I20260619 18:40:13.742214 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:13.737028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45675 (local address 127.0.0.1:34934)
0619 18:40:13.742071 (+ 5043us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:13.742124 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4942}
I20260619 18:40:13.742223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 14138
I20260619 18:40:13.748114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38743
--webserver_interface=127.0.98.62
--webserver_port=41339
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:13.874150 14672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:13.874446 14672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:13.874486 14672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:13.878520 14672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:40:13.878582 14672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:13.878600 14672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:13.878615 14672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:40:13.878638 14672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:40:13.884073 14672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:13.885332 14672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:13.886586 14672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:13.900916 14681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:13.900921 14678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:13.900975 14672 server_base.cc:1034] running on GCE node
W20260619 18:40:13.900916 14679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:13.901846 14672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:13.903216 14672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:13.904383 14672 hybrid_clock.cc:648] HybridClock initialized: now 1781894413904365 us; error 33 us; skew 500 ppm
Jun 19 18:40:13 dist-test-slave-rxv6 krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:13.907454 14672 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:40:13.908521 14672 webserver.cc:460] Webserver started at http://127.0.98.62:41339/ using document root <none> and password file <none>
I20260619 18:40:13.909176 14672 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:13.909258 14672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:13.913710 14672 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:40:13.916393 14688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:13.917989 14672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:40:13.918112 14672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "273f071c29b34623a4b14211296b834f"
format_stamp: "Formatted at 2026-06-19 18:40:12 on dist-test-slave-rxv6"
server_key: "9c1f08011dae9fcf57047f4ed3b9b395"
server_key_iv: "da188f8d7234f478fe7e09fbf816bdf8"
server_key_version: "encryptionkey@0"
I20260619 18:40:13.918833 14672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:13.942965 14672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:13.946762 14672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:13.946979 14672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:13.955130 14672 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38743
I20260619 18:40:13.955463 14740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38743 every 8 connection(s)
I20260619 18:40:13.956439 14672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:40:13.961870 14741 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:40:13.963831 14741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f: Bootstrap starting.
I20260619 18:40:13.965617 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 14672
I20260619 18:40:13.966159 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42283
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38743
--webserver_port=37291
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:40:13.983781 14741 log.cc:824] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:13.993501 14741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:40:13.993892 14741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f: Bootstrap complete.
I20260619 18:40:13.998191 14741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } }
I20260619 18:40:13.998629 14741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273f071c29b34623a4b14211296b834f, State: Initialized, Role: FOLLOWER
I20260619 18:40:13.999171 14741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } }
I20260619 18:40:13.999307 14741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:40:13.999380 14741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:40:13.999477 14741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:40:14.001713 14741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } }
I20260619 18:40:14.002081 14741 leader_election.cc:302] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 273f071c29b34623a4b14211296b834f; no voters:
I20260619 18:40:14.002421 14741 leader_election.cc:288] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:40:14.002569 14745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:40:14.002821 14745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 2 LEADER]: Becoming Leader. State: Replica: 273f071c29b34623a4b14211296b834f, State: Running, Role: LEADER
I20260619 18:40:14.003162 14745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } }
I20260619 18:40:14.003635 14741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:40:14.004230 14745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 273f071c29b34623a4b14211296b834f. Latest consensus state: current_term: 2 leader_uuid: "273f071c29b34623a4b14211296b834f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } }
I20260619 18:40:14.004351 14745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:14.004969 14750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:40:14.005048 14746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "273f071c29b34623a4b14211296b834f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } }
I20260619 18:40:14.005136 14746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:14.010236 14750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:40:14.010855 14750 catalog_manager.cc:1259] Loaded cluster ID: 6114c5649a084e07b9e367ef0981fa2a
I20260619 18:40:14.010912 14750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:40:14.012347 14750 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 18:40:14.016824 14750 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 18:40:14.022672 14750 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:40:14.023494 14750 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f: Loaded TSK: 0
I20260619 18:40:14.024930 14750 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:40:14.096007 14743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:14.096283 14743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:14.096326 14743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:14.100764 14743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:14.100835 14743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:14.100898 14743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:40:14.106526 14743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37291
--tserver_master_addrs=127.0.98.62:38743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:14.107733 14743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:14.108947 14743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:14.119621 14768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:14.119678 14771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:14.119606 14769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:14.119968 14743 server_base.cc:1034] running on GCE node
I20260619 18:40:14.120373 14743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:14.121083 14743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:14.122238 14743 hybrid_clock.cc:648] HybridClock initialized: now 1781894414122198 us; error 59 us; skew 500 ppm
Jun 19 18:40:14 dist-test-slave-rxv6 krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:14.125283 14743 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:40:14.126334 14743 webserver.cc:460] Webserver started at http://127.0.98.1:37291/ using document root <none> and password file <none>
I20260619 18:40:14.126965 14743 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:14.127053 14743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:14.131513 14743 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:40:14.133970 14778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:14.135434 14743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:40:14.135540 14743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "7fbe0fb5b8c54df5a614d2474e3b9625"
format_stamp: "Formatted at 2026-06-19 18:40:12 on dist-test-slave-rxv6"
server_key: "0b9be3c77854a8afe75872ead2d1e1a5"
server_key_iv: "a2e4cf771129f9a389e3cb6eeb203dd4"
server_key_version: "encryptionkey@0"
I20260619 18:40:14.136179 14743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:14.160679 14743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:14.164312 14743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:14.164520 14743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:14.165266 14743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:14.166530 14785 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 18:40:14.171484 14743 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 18:40:14.171561 14743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260619 18:40:14.171680 14743 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 18:40:14.173728 14743 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 18:40:14.173771 14743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 18:40:14.174258 14785 tablet_bootstrap.cc:492] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625: Bootstrap starting.
I20260619 18:40:14.184993 14743 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42283
I20260619 18:40:14.185238 14892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42283 every 8 connection(s)
I20260619 18:40:14.186205 14743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:40:14.194170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 14743
I20260619 18:40:14.194595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45675
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38743
--webserver_port=37253
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:40:14.194936 14785 log.cc:824] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:40:14 dist-test-slave-rxv6 krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:14.204231 14785 tablet_bootstrap.cc:492] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:40:14.204716 14785 tablet_bootstrap.cc:492] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625: Bootstrap complete.
I20260619 18:40:14.206107 14785 ts_tablet_manager.cc:1397] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.011s
I20260619 18:40:14.206764 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:14.188504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40459 (local address 127.0.98.62:38743)
0619 18:40:14.188971 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:14.188995 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:14.189559 (+ 564us) server_negotiation.cc:407] Connection header received
0619 18:40:14.190795 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:14.190820 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:14.191372 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:14.191815 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:14.192989 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.194576 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.196060 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.196647 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.196822 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:14.199705 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:14.199712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:14.199730 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:14.199765 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:14.202027 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:14.202612 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:14.202618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:14.202625 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:14.202793 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:14.203145 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:14.203148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:14.203150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:14.203662 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:14.203803 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:14.204912 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0619 18:40:14.205236 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":143,"threads_started":1}
I20260619 18:40:14.207262 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:14.188413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:40459)
0619 18:40:14.189122 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:14.189179 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:14.190163 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:14.191869 (+ 1706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:14.191876 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:14.192280 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:14.192808 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:14.192826 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.194787 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.194791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:14.195917 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:14.195927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.196797 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.196801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:14.196990 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:14.197590 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:14.197609 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:14.199498 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:14.202138 (+ 2640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:14.202144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:14.202157 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:14.202485 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:14.202890 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:14.202894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:14.202896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:14.203053 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:14.203773 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:14.203779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:14.204082 (+ 303us) client_negotiation.cc:769] Sending connection context
0619 18:40:14.204277 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 18:40:14.205824 (+ 1547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":192,"threads_started":1}
I20260619 18:40:14.209203 14785 raft_consensus.cc:348] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } }
I20260619 18:40:14.209677 14785 raft_consensus.cc:729] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbe0fb5b8c54df5a614d2474e3b9625, State: Initialized, Role: FOLLOWER
I20260619 18:40:14.210114 14893 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38743
I20260619 18:40:14.210295 14785 consensus_queue.cc:260] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } }
I20260619 18:40:14.210371 14893 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:14.211004 14893 heartbeater.cc:511] Master 127.0.98.62:38743 requested a full tablet report, sending...
I20260619 18:40:14.211383 14785 ts_tablet_manager.cc:1428] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260619 18:40:14.214154 14705 ts_manager.cc:194] Registered new tserver with Master: 7fbe0fb5b8c54df5a614d2474e3b9625 (127.0.98.1:42283)
I20260619 18:40:14.218956 14705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40459
I20260619 18:40:14.221714 14893 heartbeater.cc:503] Master 127.0.98.62:38743 was elected leader, sending a full tablet report...
Jun 19 18:40:14 dist-test-slave-rxv6 krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 18:40:14.228977 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:14.218329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34798 (local address 127.0.98.1:42283)
0619 18:40:14.218766 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:14.218779 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:14.219351 (+ 572us) server_negotiation.cc:407] Connection header received
0619 18:40:14.219423 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:14.219427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:14.219602 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:14.219740 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:14.220575 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.221707 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.222863 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.223261 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.223354 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:14.225700 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:14.225708 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:14.225719 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:14.225740 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:14.227197 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:14.227623 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:14.227628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:14.227630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:14.227706 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:14.227950 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:14.227953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:14.227955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:14.228210 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:14.228327 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:14.228651 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 18:40:14.228814 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":110,"threads_started":1}
I20260619 18:40:14.228977 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:14.218236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42283 (local address 127.0.0.1:34798)
0619 18:40:14.219028 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:14.219071 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:14.219234 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:14.219769 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:14.219775 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:14.220297 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:14.220444 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:14.220453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.221927 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.221931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:14.222719 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:14.222748 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.223357 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.223377 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:14.223504 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:14.224171 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:14.224183 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:14.225572 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:14.227292 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:14.227301 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:14.227307 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:14.227522 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:14.227787 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:14.227790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:14.227792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:14.227874 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:14.228314 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:14.228321 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:14.228441 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 18:40:14.228560 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 18:40:14.228780 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":120,"threads_started":1}
I20260619 18:40:14.230131 14827 tablet_service.cc:1514] Processing DeleteTablet for tablet 231a0fdd0df24e68971cada47fd260ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34798
I20260619 18:40:14.231274 14902 tablet_replica.cc:330] stopping tablet replica
I20260619 18:40:14.231643 14902 raft_consensus.cc:2229] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:14.231930 14902 raft_consensus.cc:2258] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:14.233047 14902 ts_tablet_manager.cc:1905] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:14.237259 14902 ts_tablet_manager.cc:1918] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:40:14.237368 14902 log.cc:1196] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal/wals/231a0fdd0df24e68971cada47fd260ba
I20260619 18:40:14.237650 14902 ts_tablet_manager.cc:1939] T 231a0fdd0df24e68971cada47fd260ba P 7fbe0fb5b8c54df5a614d2474e3b9625: Deleting consensus metadata
I20260619 18:40:14.239686 14691 catalog_manager.cc:4841] TS 7fbe0fb5b8c54df5a614d2474e3b9625 (127.0.98.1:42283): tablet 231a0fdd0df24e68971cada47fd260ba (table test-table [id=405f3bbd265742ba9c458363a8988380]) successfully deleted
W20260619 18:40:14.326323 14898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:14.326664 14898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:14.326723 14898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:14.330852 14898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:14.330912 14898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:14.330977 14898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:40:14.336526 14898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37253
--tserver_master_addrs=127.0.98.62:38743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:14.337880 14898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:14.339182 14898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:14.350301 14908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:14.350286 14907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:14.350359 14910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:14.350549 14898 server_base.cc:1034] running on GCE node
I20260619 18:40:14.351011 14898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:14.351776 14898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:14.352946 14898 hybrid_clock.cc:648] HybridClock initialized: now 1781894414352922 us; error 44 us; skew 500 ppm
Jun 19 18:40:14 dist-test-slave-rxv6 krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:14.356004 14898 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:40:14.357015 14898 webserver.cc:460] Webserver started at http://127.0.98.2:37253/ using document root <none> and password file <none>
I20260619 18:40:14.357650 14898 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:14.357735 14898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:14.362157 14898 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:40:14.364506 14917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:14.365865 14898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:40:14.366006 14898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "a8db4a68a7e648648e189299751e2263"
format_stamp: "Formatted at 2026-06-19 18:40:13 on dist-test-slave-rxv6"
server_key: "0fd2acb7d03eee65d6d8f41478421e07"
server_key_iv: "e240b15ed1a85d8c5696b8f65616001b"
server_key_version: "encryptionkey@0"
I20260619 18:40:14.366605 14898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:14.377593 14898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:14.380756 14898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:14.380936 14898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:14.381572 14898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:14.382786 14924 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 18:40:14.387705 14898 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 18:40:14.387765 14898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 18:40:14.387828 14898 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 18:40:14.389707 14898 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 18:40:14.389748 14898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 18:40:14.390093 14924 tablet_bootstrap.cc:492] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: Bootstrap starting.
I20260619 18:40:14.401669 14898 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45675
I20260619 18:40:14.401688 15031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45675 every 8 connection(s)
I20260619 18:40:14.403097 14898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:40:14.413429 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 14898
I20260619 18:40:14.414001 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46683
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38743
--webserver_port=44413
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:40:14.414428 14924 log.cc:824] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:40:14 dist-test-slave-rxv6 krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:14.419798 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:14.405689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45437 (local address 127.0.98.62:38743)
0619 18:40:14.405865 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:14.405870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:14.406790 (+ 920us) server_negotiation.cc:407] Connection header received
0619 18:40:14.408275 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:14.408278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:14.408339 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:14.408419 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:14.410352 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.410951 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.412852 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.413098 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.413192 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:14.416057 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:14.416061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:14.416064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:14.416097 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:14.417660 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:14.418205 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:14.418210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:14.418211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:14.418296 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:14.418610 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:14.418613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:14.418615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:14.418825 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:14.418922 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:14.419464 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 18:40:14.419613 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:40:14.420693 15034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:14.405943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:45437)
0619 18:40:14.406552 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:14.406646 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:14.407997 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:14.408691 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:14.408702 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:14.409292 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:14.410165 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:14.410197 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.411097 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.411100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:14.412715 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:14.412725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.413245 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.413249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:14.413431 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:14.414094 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:14.414111 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:14.415873 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:14.417770 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:14.417777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:14.417793 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:14.418103 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:14.418376 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:14.418379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:14.418381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:14.418515 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:14.418972 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:14.418977 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:14.419219 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 18:40:14.419385 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 18:40:14.419724 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":168,"threads_started":1}
I20260619 18:40:14.421861 15032 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38743
I20260619 18:40:14.422132 15032 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:14.422726 15032 heartbeater.cc:511] Master 127.0.98.62:38743 requested a full tablet report, sending...
I20260619 18:40:14.424677 14705 ts_manager.cc:194] Registered new tserver with Master: a8db4a68a7e648648e189299751e2263 (127.0.98.2:45675)
I20260619 18:40:14.424839 14924 tablet_bootstrap.cc:492] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:40:14.425278 14924 tablet_bootstrap.cc:492] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: Bootstrap complete.
I20260619 18:40:14.425985 14705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45437
I20260619 18:40:14.426632 14924 ts_tablet_manager.cc:1397] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.017s
I20260619 18:40:14.429944 14924 raft_consensus.cc:348] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } }
I20260619 18:40:14.430512 14924 raft_consensus.cc:729] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8db4a68a7e648648e189299751e2263, State: Initialized, Role: FOLLOWER
I20260619 18:40:14.431128 14924 consensus_queue.cc:260] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } }
I20260619 18:40:14.431687 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:14.426008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.0.98.2:45675)
0619 18:40:14.426438 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:14.426452 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:14.426474 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:40:14.427937 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:14.427943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:14.428111 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:14.428307 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:14.428531 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.429325 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.430471 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.431074 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.431170 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:14.431318 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:14.431474 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 18:40:14.431561 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":108,"threads_started":1}
I20260619 18:40:14.431685 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:14.425614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45675 (local address 127.0.0.1:34938)
0619 18:40:14.425743 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:14.425759 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:14.425843 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:14.428303 (+ 2460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:14.428307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:14.428327 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:14.428429 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:14.428436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.429423 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.429427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:14.430361 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:14.430371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.431186 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.431190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:14.431295 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:14.431367 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 18:40:14.431438 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 18:40:14.431545 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 18:40:14.432492 14924 ts_tablet_manager.cc:1428] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260619 18:40:14.433233 15032 heartbeater.cc:503] Master 127.0.98.62:38743 was elected leader, sending a full tablet report...
I20260619 18:40:14.433085 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet 231a0fdd0df24e68971cada47fd260ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34938
I20260619 18:40:14.434000 15039 tablet_replica.cc:330] stopping tablet replica
I20260619 18:40:14.434372 15039 raft_consensus.cc:2229] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:14.434653 15039 raft_consensus.cc:2258] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:14.436199 15039 ts_tablet_manager.cc:1905] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:14.440380 15039 ts_tablet_manager.cc:1918] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:40:14.440464 15039 log.cc:1196] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/231a0fdd0df24e68971cada47fd260ba
I20260619 18:40:14.440696 15039 ts_tablet_manager.cc:1939] T 231a0fdd0df24e68971cada47fd260ba P a8db4a68a7e648648e189299751e2263: Deleting consensus metadata
I20260619 18:40:14.442433 14692 catalog_manager.cc:4841] TS a8db4a68a7e648648e189299751e2263 (127.0.98.2:45675): tablet 231a0fdd0df24e68971cada47fd260ba (table test-table [id=405f3bbd265742ba9c458363a8988380]) successfully deleted
W20260619 18:40:14.545300 15036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:14.545617 15036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:14.545687 15036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:14.549880 15036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:14.549981 15036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:14.550071 15036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:40:14.556005 15036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44413
--tserver_master_addrs=127.0.98.62:38743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:14.557648 15036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:14.558905 15036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:14.569392 15045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:14.569423 15047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:14.569572 15036 server_base.cc:1034] running on GCE node
W20260619 18:40:14.569392 15044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:14.570163 15036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:14.570917 15036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:14.572072 15036 hybrid_clock.cc:648] HybridClock initialized: now 1781894414572053 us; error 30 us; skew 500 ppm
Jun 19 18:40:14 dist-test-slave-rxv6 krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:14.574963 15036 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:40:14.575939 15036 webserver.cc:460] Webserver started at http://127.0.98.3:44413/ using document root <none> and password file <none>
I20260619 18:40:14.576561 15036 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:14.576643 15036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:14.580852 15036 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:40:14.583110 15054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:14.584426 15036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:40:14.584592 15036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "28bd1f116dca4597bdd411bc83987f53"
format_stamp: "Formatted at 2026-06-19 18:40:13 on dist-test-slave-rxv6"
server_key: "310c9e7772acb48a8d8f2fd85f853519"
server_key_iv: "650991851c9c50b0ed2e4d938584383b"
server_key_version: "encryptionkey@0"
I20260619 18:40:14.585144 15036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:14.609736 15036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:14.613098 15036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:14.613286 15036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:14.613932 15036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:14.614820 15036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:14.614868 15036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:14.614921 15036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:14.614951 15036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:14.624017 15036 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46683
I20260619 18:40:14.624085 15167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46683 every 8 connection(s)
I20260619 18:40:14.625355 15036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:40:14.632274 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 15036
Jun 19 18:40:14 dist-test-slave-rxv6 krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:14.639534 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:14.627540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34559 (local address 127.0.98.62:38743)
0619 18:40:14.627726 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:14.627730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:14.628771 (+ 1041us) server_negotiation.cc:407] Connection header received
0619 18:40:14.629695 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:14.629699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:14.629764 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:14.629839 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:14.631080 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.631625 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.632944 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.633275 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.633390 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:14.635879 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:14.635883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:14.635889 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:14.635913 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:14.637288 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:14.637783 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:14.637786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:14.637787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:14.637864 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:14.638271 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:14.638274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:14.638275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:14.638500 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:14.638604 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:14.639231 (+ 627us) server_negotiation.cc:299] Negotiation successful
0619 18:40:14.639350 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 18:40:14.640587 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:14.627938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:34559)
0619 18:40:14.628617 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:14.628658 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:14.629475 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:14.630072 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:14.630079 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:14.630440 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:14.630927 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:14.630951 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.631739 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.631742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:14.632799 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:14.632810 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.633440 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.633445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:14.633632 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:14.634198 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:14.634213 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:14.635708 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:14.637391 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:14.637397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:14.637413 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:14.637685 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:14.637987 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:14.637990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:14.637992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:14.638157 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:14.638613 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:14.638620 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:14.638935 (+ 315us) client_negotiation.cc:769] Sending connection context
0619 18:40:14.639155 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 18:40:14.639489 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":108,"threads_started":1}
I20260619 18:40:14.641245 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:14.632927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48542 (local address 127.0.98.62:38743)
0619 18:40:14.633631 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:14.633634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:14.633644 (+ 10us) server_negotiation.cc:407] Connection header received
0619 18:40:14.633672 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:14.633673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:14.633704 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:14.633778 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:14.634538 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.635233 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.636133 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.636411 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.636522 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:14.637692 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:14.637695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:14.637698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:14.637729 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:14.639278 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:14.639974 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:14.639977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:14.639979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:14.640080 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:14.640396 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:14.640399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:14.640401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:14.640592 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:14.640890 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:14.640966 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 18:40:14.641098 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":109,"threads_started":1}
I20260619 18:40:14.641999 15168 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38743
I20260619 18:40:14.642248 15168 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:14.642825 15168 heartbeater.cc:511] Master 127.0.98.62:38743 requested a full tablet report, sending...
I20260619 18:40:14.643918 14705 ts_manager.cc:194] Registered new tserver with Master: 28bd1f116dca4597bdd411bc83987f53 (127.0.98.3:46683)
I20260619 18:40:14.644475 14705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34559
I20260619 18:40:14.654973 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:40:14.659427 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:14.655634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48548 (local address 127.0.98.62:38743)
0619 18:40:14.655782 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:14.655787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:14.655970 (+ 183us) server_negotiation.cc:407] Connection header received
0619 18:40:14.656158 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:14.656160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:14.656265 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:14.656353 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:14.656643 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.657163 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.658029 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.658262 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.658346 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:14.658646 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:14.659071 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:14.659180 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:14.659239 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 18:40:14.659298 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:40:14.660105 14705 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:40:14.663746 14705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:40:14.671554 14966 tablet_service.cc:1467] Processing CreateTablet for tablet 018c8fe4d76b4a218c0171e2cebfa8c7 (DEFAULT_TABLE table=test-table [id=e5ff9a6e97104f67839609176fdafe4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:14.671540 14827 tablet_service.cc:1467] Processing CreateTablet for tablet 018c8fe4d76b4a218c0171e2cebfa8c7 (DEFAULT_TABLE table=test-table [id=e5ff9a6e97104f67839609176fdafe4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:14.672147 14966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018c8fe4d76b4a218c0171e2cebfa8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:14.672166 14827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018c8fe4d76b4a218c0171e2cebfa8c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:14.675001 14924 tablet_bootstrap.cc:492] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263: Bootstrap starting.
I20260619 18:40:14.675053 14785 tablet_bootstrap.cc:492] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625: Bootstrap starting.
I20260619 18:40:14.675890 14785 tablet_bootstrap.cc:654] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:14.675902 14924 tablet_bootstrap.cc:654] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:14.676851 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:14.670722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46683 (local address 127.0.0.1:33098)
0619 18:40:14.670825 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:14.670840 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:14.670911 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:14.672624 (+ 1713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:14.672627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:14.672648 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:14.672749 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:14.672756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.674281 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.674285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:14.675357 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:14.675369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.676484 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.676488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:14.676566 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:14.676591 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 18:40:14.676646 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 18:40:14.676725 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 18:40:14.677074 14924 tablet_bootstrap.cc:492] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263: No bootstrap required, opened a new log
I20260619 18:40:14.677160 14924 ts_tablet_manager.cc:1397] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 18:40:14.677513 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:14.670870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33098 (local address 127.0.98.3:46683)
0619 18:40:14.671998 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:14.672009 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:14.672035 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:40:14.672114 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:14.672120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:14.672459 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:14.672633 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:14.672892 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.674141 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.675501 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:14.676330 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:14.677021 (+ 691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:14.677183 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:14.677243 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 18:40:14.677341 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":220,"threads_started":1}
I20260619 18:40:14.677963 14785 tablet_bootstrap.cc:492] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625: No bootstrap required, opened a new log
I20260619 18:40:14.677881 14924 raft_consensus.cc:348] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } }
I20260619 18:40:14.678032 14924 raft_consensus.cc:374] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:14.678048 14785 ts_tablet_manager.cc:1397] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 18:40:14.678090 14924 raft_consensus.cc:729] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8db4a68a7e648648e189299751e2263, State: Initialized, Role: FOLLOWER
I20260619 18:40:14.678215 14924 consensus_queue.cc:260] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } }
I20260619 18:40:14.678462 14924 ts_tablet_manager.cc:1428] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 18:40:14.678472 14785 raft_consensus.cc:348] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } }
I20260619 18:40:14.678581 14785 raft_consensus.cc:374] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:14.678615 14785 raft_consensus.cc:729] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbe0fb5b8c54df5a614d2474e3b9625, State: Initialized, Role: FOLLOWER
I20260619 18:40:14.678732 14785 consensus_queue.cc:260] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } }
I20260619 18:40:14.678973 14785 ts_tablet_manager.cc:1428] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 18:40:14.681082 15102 tablet_service.cc:1467] Processing CreateTablet for tablet 018c8fe4d76b4a218c0171e2cebfa8c7 (DEFAULT_TABLE table=test-table [id=e5ff9a6e97104f67839609176fdafe4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:14.682135 15102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 018c8fe4d76b4a218c0171e2cebfa8c7. 1 dirs total, 0 dirs full, 0 dirs failed
W20260619 18:40:14.687474 14894 tablet.cc:2369] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:40:14.688917 15180 tablet_bootstrap.cc:492] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53: Bootstrap starting.
I20260619 18:40:14.691522 15180 tablet_bootstrap.cc:654] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:14.692291 15180 log.cc:824] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:14.694132 15180 tablet_bootstrap.cc:492] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53: No bootstrap required, opened a new log
I20260619 18:40:14.694324 15180 ts_tablet_manager.cc:1397] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:40:14.697450 15180 raft_consensus.cc:348] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } }
I20260619 18:40:14.697696 15180 raft_consensus.cc:374] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:14.697753 15180 raft_consensus.cc:729] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28bd1f116dca4597bdd411bc83987f53, State: Initialized, Role: FOLLOWER
I20260619 18:40:14.698235 15180 consensus_queue.cc:260] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } }
I20260619 18:40:14.698859 15168 heartbeater.cc:503] Master 127.0.98.62:38743 was elected leader, sending a full tablet report...
I20260619 18:40:14.699123 15180 ts_tablet_manager.cc:1428] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 18:40:14.876453 15169 tablet.cc:2369] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:40:14.904520 15033 tablet.cc:2369] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:40:15.002341 15182 raft_consensus.cc:482] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:15.002554 15182 raft_consensus.cc:504] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } }
I20260619 18:40:15.003854 15182 leader_election.cc:288] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8db4a68a7e648648e189299751e2263 (127.0.98.2:45675), 7fbe0fb5b8c54df5a614d2474e3b9625 (127.0.98.1:42283)
I20260619 18:40:15.007618 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:15.004112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45675 (local address 127.0.98.3:57969)
0619 18:40:15.004252 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:15.004268 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:15.004366 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:15.004788 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:15.004791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:15.004810 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:15.004928 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:15.004933 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:15.005707 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:15.005710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:15.006507 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:15.006516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:15.007198 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:15.007202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:15.007310 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:15.007331 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:40:15.007395 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 18:40:15.007468 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 18:40:15.007622 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:15.004290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57969 (local address 127.0.98.2:45675)
0619 18:40:15.004538 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:15.004541 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:15.004556 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:40:15.004615 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:15.004618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:15.004671 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:15.004782 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:15.005039 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:15.005602 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:15.006639 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:15.007086 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:15.007181 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:15.007250 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:15.007424 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 18:40:15.007501 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
I20260619 18:40:15.008047 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:15.004211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48123 (local address 127.0.98.1:42283)
0619 18:40:15.004541 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:15.004546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:15.004894 (+ 348us) server_negotiation.cc:407] Connection header received
0619 18:40:15.005022 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:15.005025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:15.005083 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:15.005168 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:15.005412 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:15.006080 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:15.007070 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:15.007529 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:15.007623 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:15.007809 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:15.007844 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 18:40:15.007913 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":133,"threads_started":1}
I20260619 18:40:15.008044 15185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:15.004112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42283 (local address 127.0.98.3:48123)
0619 18:40:15.004810 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:15.004824 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:15.004915 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:15.005186 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:15.005188 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:15.005203 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:15.005313 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:15.005318 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:15.006187 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:15.006191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:15.006949 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:15.006957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:15.007655 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:15.007658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:15.007745 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:15.007764 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 18:40:15.007820 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 18:40:15.007888 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":323,"thread_start_us":168,"threads_started":1}
I20260619 18:40:15.008253 14984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018c8fe4d76b4a218c0171e2cebfa8c7" candidate_uuid: "28bd1f116dca4597bdd411bc83987f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8db4a68a7e648648e189299751e2263" is_pre_election: true
I20260619 18:40:15.008606 14984 raft_consensus.cc:2454] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28bd1f116dca4597bdd411bc83987f53 in term 0.
I20260619 18:40:15.008591 14847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018c8fe4d76b4a218c0171e2cebfa8c7" candidate_uuid: "28bd1f116dca4597bdd411bc83987f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" is_pre_election: true
I20260619 18:40:15.008839 14847 raft_consensus.cc:2454] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28bd1f116dca4597bdd411bc83987f53 in term 0.
I20260619 18:40:15.009130 15058 leader_election.cc:302] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28bd1f116dca4597bdd411bc83987f53, a8db4a68a7e648648e189299751e2263; no voters:
I20260619 18:40:15.009423 15182 raft_consensus.cc:2781] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:40:15.009508 15182 raft_consensus.cc:482] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:40:15.009547 15182 raft_consensus.cc:3037] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:15.010576 15182 raft_consensus.cc:504] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } }
I20260619 18:40:15.010895 15182 leader_election.cc:288] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [CANDIDATE]: Term 1 election: Requested vote from peers a8db4a68a7e648648e189299751e2263 (127.0.98.2:45675), 7fbe0fb5b8c54df5a614d2474e3b9625 (127.0.98.1:42283)
I20260619 18:40:15.011232 14984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018c8fe4d76b4a218c0171e2cebfa8c7" candidate_uuid: "28bd1f116dca4597bdd411bc83987f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8db4a68a7e648648e189299751e2263"
I20260619 18:40:15.011252 14847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "018c8fe4d76b4a218c0171e2cebfa8c7" candidate_uuid: "28bd1f116dca4597bdd411bc83987f53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625"
I20260619 18:40:15.011366 14847 raft_consensus.cc:3037] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:15.011368 14984 raft_consensus.cc:3037] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:15.012552 14984 raft_consensus.cc:2454] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28bd1f116dca4597bdd411bc83987f53 in term 1.
I20260619 18:40:15.012552 14847 raft_consensus.cc:2454] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28bd1f116dca4597bdd411bc83987f53 in term 1.
I20260619 18:40:15.012849 15057 leader_election.cc:302] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28bd1f116dca4597bdd411bc83987f53, 7fbe0fb5b8c54df5a614d2474e3b9625; no voters:
I20260619 18:40:15.013058 15182 raft_consensus.cc:2781] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:15.013242 15182 raft_consensus.cc:686] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [term 1 LEADER]: Becoming Leader. State: Replica: 28bd1f116dca4597bdd411bc83987f53, State: Running, Role: LEADER
I20260619 18:40:15.013584 15182 consensus_queue.cc:237] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } }
I20260619 18:40:15.016700 14705 catalog_manager.cc:5495] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28bd1f116dca4597bdd411bc83987f53 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "28bd1f116dca4597bdd411bc83987f53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } health_report { overall_health: UNKNOWN } } }
I20260619 18:40:15.041661 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:15.038247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.0.98.3:46683)
0619 18:40:15.038380 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:15.038385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:15.038398 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:40:15.038488 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:15.038491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:15.038551 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:15.038639 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:15.038855 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:15.039378 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:15.040250 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:15.040484 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:15.040573 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:15.040843 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:15.041286 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:15.041388 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:15.041464 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 18:40:15.041522 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:40:15.047801 14984 raft_consensus.cc:1261] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263 [term 1 FOLLOWER]: Refusing update from remote peer 28bd1f116dca4597bdd411bc83987f53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:15.047807 14847 raft_consensus.cc:1261] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Refusing update from remote peer 28bd1f116dca4597bdd411bc83987f53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:15.048482 15186 consensus_queue.cc:1035] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:15.048659 15182 consensus_queue.cc:1035] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:15.054296 15191 mvcc.cc:205] Tried to move back new op lower bound from 7298639524028284928 to 7298639523898134528. Current Snapshot: MvccSnapshot[applied={T|T < 7298639524028284928}]
I20260619 18:40:15.062659 14704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:15.062844 14704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:15.065171 14704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f: Sending DeleteTablet for 3 replicas of tablet 018c8fe4d76b4a218c0171e2cebfa8c7
I20260619 18:40:15.065941 14827 tablet_service.cc:1514] Processing DeleteTablet for tablet 018c8fe4d76b4a218c0171e2cebfa8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34798
I20260619 18:40:15.065888 15102 tablet_service.cc:1514] Processing DeleteTablet for tablet 018c8fe4d76b4a218c0171e2cebfa8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33098
I20260619 18:40:15.066257 14966 tablet_service.cc:1514] Processing DeleteTablet for tablet 018c8fe4d76b4a218c0171e2cebfa8c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34938
I20260619 18:40:15.066494 15200 tablet_replica.cc:330] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53: stopping tablet replica
I20260619 18:40:15.066529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 14743
I20260619 18:40:15.066576 15201 tablet_replica.cc:330] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263: stopping tablet replica
I20260619 18:40:15.067148 15201 raft_consensus.cc:2229] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:15.067240 15200 raft_consensus.cc:2229] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:40:15.067276 15201 raft_consensus.cc:2258] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:15.067618 15200 raft_consensus.cc:2258] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:15.068176 15201 ts_tablet_manager.cc:1905] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:15.069063 15200 ts_tablet_manager.cc:1905] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:15.070705 15201 ts_tablet_manager.cc:1918] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:40:15.070794 15201 log.cc:1196] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/018c8fe4d76b4a218c0171e2cebfa8c7
I20260619 18:40:15.071142 15201 ts_tablet_manager.cc:1939] T 018c8fe4d76b4a218c0171e2cebfa8c7 P a8db4a68a7e648648e189299751e2263: Deleting consensus metadata
I20260619 18:40:15.071841 15200 ts_tablet_manager.cc:1918] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:40:15.071835 14692 catalog_manager.cc:4841] TS a8db4a68a7e648648e189299751e2263 (127.0.98.2:45675): tablet 018c8fe4d76b4a218c0171e2cebfa8c7 (table test-table [id=e5ff9a6e97104f67839609176fdafe4c]) successfully deleted
I20260619 18:40:15.071934 15200 log.cc:1196] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/018c8fe4d76b4a218c0171e2cebfa8c7
I20260619 18:40:15.072243 15200 ts_tablet_manager.cc:1939] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 28bd1f116dca4597bdd411bc83987f53: Deleting consensus metadata
I20260619 18:40:15.073223 14692 catalog_manager.cc:4841] TS 28bd1f116dca4597bdd411bc83987f53 (127.0.98.3:46683): tablet 018c8fe4d76b4a218c0171e2cebfa8c7 (table test-table [id=e5ff9a6e97104f67839609176fdafe4c]) successfully deleted
W20260619 18:40:15.075920 14691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42283 (error 108)
I20260619 18:40:15.076483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 14898
I20260619 18:40:15.077026 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:15.076712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42283 (local address 127.0.0.1:34808)
0619 18:40:15.076869 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:15.076930 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260619 18:40:15.077298 14691 catalog_manager.cc:4584] TS 7fbe0fb5b8c54df5a614d2474e3b9625 (127.0.98.1:42283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 018c8fe4d76b4a218c0171e2cebfa8c7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42283: connect: Connection refused (error 111)
I20260619 18:40:15.081609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 15036
I20260619 18:40:15.086617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 14672
I20260619 18:40:15.091735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38743
--webserver_interface=127.0.98.62
--webserver_port=41339
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:15.219967 15203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:15.220239 15203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:15.220297 15203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:15.224335 15203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:40:15.224399 15203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:15.224423 15203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:15.224444 15203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:40:15.224464 15203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:40:15.229689 15203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41339
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:15.230715 15203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:15.231688 15203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:15.240226 15208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:15.240270 15211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:15.240233 15209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:15.240497 15203 server_base.cc:1034] running on GCE node
I20260619 18:40:15.241015 15203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:15.242200 15203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:15.243364 15203 hybrid_clock.cc:648] HybridClock initialized: now 1781894415243345 us; error 31 us; skew 500 ppm
Jun 19 18:40:15 dist-test-slave-rxv6 krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:15.246007 15203 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:40:15.246881 15203 webserver.cc:460] Webserver started at http://127.0.98.62:41339/ using document root <none> and password file <none>
I20260619 18:40:15.247409 15203 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:15.247471 15203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:15.251184 15203 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:40:15.253260 15218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:15.254490 15203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:40:15.254604 15203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "273f071c29b34623a4b14211296b834f"
format_stamp: "Formatted at 2026-06-19 18:40:12 on dist-test-slave-rxv6"
server_key: "9c1f08011dae9fcf57047f4ed3b9b395"
server_key_iv: "da188f8d7234f478fe7e09fbf816bdf8"
server_key_version: "encryptionkey@0"
I20260619 18:40:15.255173 15203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:15.264186 15203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:15.267151 15203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:15.267333 15203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:15.273938 15203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38743
I20260619 18:40:15.274209 15270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38743 every 8 connection(s)
I20260619 18:40:15.275008 15203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:40:15.277766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 15203
I20260619 18:40:15.278226 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42283
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38743
--webserver_port=37291
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:40:15.279526 15271 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:40:15.281139 15271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f: Bootstrap starting.
I20260619 18:40:15.298106 15271 log.cc:824] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:15.309324 15271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:40:15.309685 15271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f: Bootstrap complete.
I20260619 18:40:15.313575 15271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } }
I20260619 18:40:15.313992 15271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273f071c29b34623a4b14211296b834f, State: Initialized, Role: FOLLOWER
I20260619 18:40:15.314489 15271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } }
I20260619 18:40:15.314625 15271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:40:15.314687 15271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:40:15.314778 15271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 2 FOLLOWER]: Advancing to term 3
I20260619 18:40:15.316649 15271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } }
I20260619 18:40:15.316969 15271 leader_election.cc:302] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 273f071c29b34623a4b14211296b834f; no voters:
I20260619 18:40:15.317260 15271 leader_election.cc:288] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 18:40:15.317391 15275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 3 FOLLOWER]: Leader election won for term 3
I20260619 18:40:15.317626 15275 raft_consensus.cc:686] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [term 3 LEADER]: Becoming Leader. State: Replica: 273f071c29b34623a4b14211296b834f, State: Running, Role: LEADER
I20260619 18:40:15.317961 15275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } }
I20260619 18:40:15.318466 15271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:40:15.319419 15276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "273f071c29b34623a4b14211296b834f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } }
I20260619 18:40:15.319561 15276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:15.320261 15277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 273f071c29b34623a4b14211296b834f. Latest consensus state: current_term: 3 leader_uuid: "273f071c29b34623a4b14211296b834f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "273f071c29b34623a4b14211296b834f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38743 } } }
I20260619 18:40:15.320356 15277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 273f071c29b34623a4b14211296b834f [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:15.320345 15284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:40:15.325577 15284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:40:15.326282 15284 catalog_manager.cc:1259] Loaded cluster ID: 6114c5649a084e07b9e367ef0981fa2a
I20260619 18:40:15.326344 15284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 18:40:15.327355 15284 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd331e439ad google::LogMessage::Fail() at ??:0
@ 0x7fd331e4688a google::LogMessage::SendToLog() at ??:0
@ 0x7fd331e434de google::LogMessage::Flush() at ??:0
@ 0x7fd331e44a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd33c782fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fd33c783d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd33c780579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fd33c7c4640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd3340953d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd334095c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fd3340973f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd334086512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd3356ca6db start_thread at ??:0
@ 0x7fd33094161f clone at ??:0
W20260619 18:40:15.408355 15273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:15.408591 15273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:15.408633 15273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:15.412621 15273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:15.412673 15273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:15.412734 15273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:40:15.418262 15273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37291
--tserver_master_addrs=127.0.98.62:38743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:15.419276 15273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:15.420300 15273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:15.429694 15316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:15.429798 15314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:15.429817 15313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:15.430538 15273 server_base.cc:1034] running on GCE node
I20260619 18:40:15.430943 15273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:15.431582 15273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:15.432754 15273 hybrid_clock.cc:648] HybridClock initialized: now 1781894415432737 us; error 28 us; skew 500 ppm
Jun 19 18:40:15 dist-test-slave-rxv6 krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:15.435498 15273 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:40:15.436368 15273 webserver.cc:460] Webserver started at http://127.0.98.1:37291/ using document root <none> and password file <none>
I20260619 18:40:15.436928 15273 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:15.437000 15273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:15.441145 15273 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:40:15.443463 15323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:15.444793 15273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:40:15.444900 15273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "7fbe0fb5b8c54df5a614d2474e3b9625"
format_stamp: "Formatted at 2026-06-19 18:40:12 on dist-test-slave-rxv6"
server_key: "0b9be3c77854a8afe75872ead2d1e1a5"
server_key_iv: "a2e4cf771129f9a389e3cb6eeb203dd4"
server_key_version: "encryptionkey@0"
I20260619 18:40:15.445425 15273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:15.458879 15273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:15.462074 15273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:15.462244 15273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:15.462805 15273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:15.463881 15330 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 18:40:15.469019 15273 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 18:40:15.469097 15273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 18:40:15.469165 15273 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 18:40:15.470798 15273 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 18:40:15.470844 15273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260619 18:40:15.471194 15330 tablet_bootstrap.cc:492] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625: Bootstrap starting.
I20260619 18:40:15.480425 15273 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42283
I20260619 18:40:15.480695 15437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42283 every 8 connection(s)
I20260619 18:40:15.481418 15273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:40:15.484872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 15273
I20260619 18:40:15.485352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45675
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38743
--webserver_port=37253
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:40:15.492937 15330 log.cc:824] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:15.502058 15330 tablet_bootstrap.cc:492] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:40:15.502470 15330 tablet_bootstrap.cc:492] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625: Bootstrap complete.
I20260619 18:40:15.503775 15330 ts_tablet_manager.cc:1397] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.018s
I20260619 18:40:15.507035 15330 raft_consensus.cc:348] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } }
I20260619 18:40:15.507591 15330 raft_consensus.cc:729] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbe0fb5b8c54df5a614d2474e3b9625, State: Initialized, Role: FOLLOWER
I20260619 18:40:15.508106 15330 consensus_queue.cc:260] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } }
I20260619 18:40:15.509152 15330 ts_tablet_manager.cc:1428] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
W20260619 18:40:15.535142 15440 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:40:15.483308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:33847)
0619 18:40:15.483854 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:15.483913 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:15.484985 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:15.534409 (+ 49424us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":123,"threads_started":1}
W20260619 18:40:15.535706 15438 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38743 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38743: Client connection negotiation failed: client connection to 127.0.98.62:38743: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 18:40:15.536468 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:15.536197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:49865)
0619 18:40:15.536342 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:15.536408 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 18:40:15.537122 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:15.536920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:40217)
0619 18:40:15.537014 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:15.537072 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 18:40:15.537292 15438 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:40:15.600804 15441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:15.601047 15441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:15.601100 15441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:15.605084 15441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:15.605140 15441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:15.605196 15441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:40:15.610248 15441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37253
--tserver_master_addrs=127.0.98.62:38743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:15.611183 15441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:15.612138 15441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:15.621229 15448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:15.621299 15441 server_base.cc:1034] running on GCE node
W20260619 18:40:15.621236 15449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:15.621225 15451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:15.621842 15441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:15.622453 15441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:15.623623 15441 hybrid_clock.cc:648] HybridClock initialized: now 1781894415623604 us; error 28 us; skew 500 ppm
Jun 19 18:40:15 dist-test-slave-rxv6 krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:15.626242 15441 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:40:15.627256 15441 webserver.cc:460] Webserver started at http://127.0.98.2:37253/ using document root <none> and password file <none>
I20260619 18:40:15.628003 15441 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:15.628090 15441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:15.631930 15441 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 18:40:15.633824 15458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:15.635044 15441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:40:15.635149 15441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "a8db4a68a7e648648e189299751e2263"
format_stamp: "Formatted at 2026-06-19 18:40:13 on dist-test-slave-rxv6"
server_key: "0fd2acb7d03eee65d6d8f41478421e07"
server_key_iv: "e240b15ed1a85d8c5696b8f65616001b"
server_key_version: "encryptionkey@0"
I20260619 18:40:15.635659 15441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:15.650345 15441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:15.653183 15441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:15.653358 15441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:15.653971 15441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:15.654826 15441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:15.654874 15441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:15.654927 15441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:15.654953 15441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:15.663554 15441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45675
I20260619 18:40:15.663610 15571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45675 every 8 connection(s)
I20260619 18:40:15.664498 15441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:40:15.668054 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:15.666506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:46005)
0619 18:40:15.666931 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:15.667404 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
W20260619 18:40:15.668504 15572 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38743 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38743: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
I20260619 18:40:15.669044 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:15.668855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:47979)
0619 18:40:15.668950 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:15.668993 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 18:40:15.669601 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:15.669420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:46823)
0619 18:40:15.669509 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:15.669550 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 18:40:15.669740 15572 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 18:40:15.672983 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 15441
I20260619 18:40:15.673308 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46683
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38743
--webserver_port=44413
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:15.777359 15575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:15.777590 15575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:15.777637 15575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:15.781538 15575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:15.781610 15575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:15.781667 15575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:40:15.786744 15575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44413
--tserver_master_addrs=127.0.98.62:38743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:15.787672 15575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:15.788601 15575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:15.798498 15581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:15.798714 15583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:15.798511 15580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:15.798983 15575 server_base.cc:1034] running on GCE node
I20260619 18:40:15.799450 15575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:15.800220 15575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:15.801399 15575 hybrid_clock.cc:648] HybridClock initialized: now 1781894415801385 us; error 34 us; skew 500 ppm
Jun 19 18:40:15 dist-test-slave-rxv6 krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:15.804132 15575 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:40:15.804947 15575 webserver.cc:460] Webserver started at http://127.0.98.3:44413/ using document root <none> and password file <none>
I20260619 18:40:15.805445 15575 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:15.805500 15575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:15.809176 15575 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:40:15.811132 15590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:15.812270 15575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:40:15.812368 15575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "28bd1f116dca4597bdd411bc83987f53"
format_stamp: "Formatted at 2026-06-19 18:40:13 on dist-test-slave-rxv6"
server_key: "310c9e7772acb48a8d8f2fd85f853519"
server_key_iv: "650991851c9c50b0ed2e4d938584383b"
server_key_version: "encryptionkey@0"
I20260619 18:40:15.812888 15575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:15.831909 15575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:15.834609 15575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:15.834758 15575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:15.835269 15575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:15.836014 15575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:15.836055 15575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:15.836092 15575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:15.836110 15575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:15.844520 15575 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46683
I20260619 18:40:15.844545 15703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46683 every 8 connection(s)
I20260619 18:40:15.845481 15575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:40:15.848773 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 15575
I20260619 18:40:15.849210 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:15.847504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:52767)
0619 18:40:15.847938 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:15.848452 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":97,"threads_started":1}
W20260619 18:40:15.849617 15704 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38743 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38743: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
I20260619 18:40:15.850195 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:15.850005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:53185)
0619 18:40:15.850100 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:15.850141 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 18:40:15.850719 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:15.850514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:51049)
0619 18:40:15.850633 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:15.850671 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 18:40:15.850852 15704 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 18:40:16.538290 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:16.537851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:44655)
0619 18:40:16.538116 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:16.538172 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":106,"threads_started":1}
I20260619 18:40:16.670784 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:16.670317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:37527)
0619 18:40:16.670607 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:16.670669 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260619 18:40:16.851817 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:16.851366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:39133)
0619 18:40:16.851648 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:16.851704 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
I20260619 18:40:17.051265 15711 raft_consensus.cc:482] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:17.051537 15711 raft_consensus.cc:504] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } }
I20260619 18:40:17.052879 15711 leader_election.cc:288] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28bd1f116dca4597bdd411bc83987f53 (127.0.98.3:46683), a8db4a68a7e648648e189299751e2263 (127.0.98.2:45675)
Jun 19 18:40:17 dist-test-slave-rxv6 krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 19 18:40:17 dist-test-slave-rxv6 krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260619 18:40:17.067605 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:17.053208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46683 (local address 127.0.98.1:41059)
0619 18:40:17.053818 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:17.053833 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:17.054065 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:17.055809 (+ 1744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:17.055815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:17.056421 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:17.056988 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:17.057005 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:17.058554 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:17.058557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:17.059578 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:17.059588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:17.060273 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:17.060277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:17.060478 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:17.061189 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:17.061206 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:17.063134 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:17.065166 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:17.065172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:17.065188 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:17.065458 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:17.065773 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:17.065776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:17.065778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:17.065902 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:17.066567 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:17.066574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:17.066829 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 18:40:17.066995 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 18:40:17.067352 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":229,"spinlock_wait_cycles":23808,"thread_start_us":92,"threads_started":1}
I20260619 18:40:17.067623 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:17.053367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41059 (local address 127.0.98.3:46683)
0619 18:40:17.053724 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:17.053759 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:17.053963 (+ 204us) server_negotiation.cc:407] Connection header received
0619 18:40:17.054983 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:17.054989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:17.055410 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:17.055731 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:17.057161 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:17.058413 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:17.059687 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:17.060163 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:17.060313 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:17.063304 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:17.063309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:17.063320 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:17.063349 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:17.065057 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:17.065563 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:17.065568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:17.065572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:17.065688 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:17.066004 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:17.066007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:17.066009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:17.066444 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:17.066567 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:17.067143 (+ 576us) server_negotiation.cc:299] Negotiation successful
0619 18:40:17.067414 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":95,"threads_started":1}
I20260619 18:40:17.067651 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:17.053616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45675 (local address 127.0.98.1:40239)
0619 18:40:17.053865 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:17.053876 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:17.054065 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:17.055809 (+ 1744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:17.055815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:17.056421 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:17.056988 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:17.057004 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:17.058594 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:17.058597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:17.059594 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:17.059604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:17.060278 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:17.060282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:17.060477 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:17.061232 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:17.061243 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:17.062887 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:17.065211 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:17.065215 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:17.065217 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:17.065465 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:17.065799 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:17.065802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:17.065805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:17.065939 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:17.066568 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:17.066575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:17.066829 (+ 254us) client_negotiation.cc:769] Sending connection context
0619 18:40:17.066998 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 18:40:17.067440 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":357,"spinlock_wait_cycles":67456,"thread_start_us":72,"threads_started":1}
I20260619 18:40:17.068252 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:17.053839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40239 (local address 127.0.98.2:45675)
0619 18:40:17.054192 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:17.054219 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:17.054252 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:40:17.054980 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:17.054987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:17.055409 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:17.055732 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:17.057163 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:17.058460 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:17.059708 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:17.060170 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:17.060313 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:17.063140 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:17.063146 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:17.063160 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:17.063194 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:17.065093 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:17.065575 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:17.065581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:17.065585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:17.065706 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:17.066033 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:17.066037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:17.066039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:17.066446 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:17.066579 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:17.067150 (+ 571us) server_negotiation.cc:299] Negotiation successful
0619 18:40:17.067440 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":164,"threads_started":1}
W20260619 18:40:17.070708 15327 leader_election.cc:341] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8db4a68a7e648648e189299751e2263 (127.0.98.2:45675): Not found: Tablet not found: 018c8fe4d76b4a218c0171e2cebfa8c7
W20260619 18:40:17.070910 15327 leader_election.cc:341] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28bd1f116dca4597bdd411bc83987f53 (127.0.98.3:46683): Not found: Tablet not found: 018c8fe4d76b4a218c0171e2cebfa8c7
I20260619 18:40:17.070972 15327 leader_election.cc:302] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fbe0fb5b8c54df5a614d2474e3b9625; no voters: 28bd1f116dca4597bdd411bc83987f53, a8db4a68a7e648648e189299751e2263
I20260619 18:40:17.071269 15711 raft_consensus.cc:2726] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:40:17.539408 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:17.539089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:58895)
0619 18:40:17.539230 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:17.539292 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 18:40:17.671962 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:17.671543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:35517)
0619 18:40:17.671807 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:17.671861 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":97,"threads_started":1}
I20260619 18:40:17.852981 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:17.852526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:49683)
0619 18:40:17.852799 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:17.852869 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":86,"threads_started":1}
I20260619 18:40:18.540817 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:18.540254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:57145)
0619 18:40:18.540613 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:18.540685 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":97,"threads_started":1}
I20260619 18:40:18.673213 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:18.672759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:38125)
0619 18:40:18.673040 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:18.673100 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
I20260619 18:40:18.854205 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:18.853732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:39425)
0619 18:40:18.854031 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:18.854090 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260619 18:40:19.062112 15721 raft_consensus.cc:482] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:19.062271 15721 raft_consensus.cc:504] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } }
I20260619 18:40:19.062743 15721 leader_election.cc:288] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28bd1f116dca4597bdd411bc83987f53 (127.0.98.3:46683), a8db4a68a7e648648e189299751e2263 (127.0.98.2:45675)
W20260619 18:40:19.063606 15327 leader_election.cc:341] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28bd1f116dca4597bdd411bc83987f53 (127.0.98.3:46683): Not found: Tablet not found: 018c8fe4d76b4a218c0171e2cebfa8c7
W20260619 18:40:19.063740 15327 leader_election.cc:341] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8db4a68a7e648648e189299751e2263 (127.0.98.2:45675): Not found: Tablet not found: 018c8fe4d76b4a218c0171e2cebfa8c7
I20260619 18:40:19.063794 15327 leader_election.cc:302] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fbe0fb5b8c54df5a614d2474e3b9625; no voters: 28bd1f116dca4597bdd411bc83987f53, a8db4a68a7e648648e189299751e2263
I20260619 18:40:19.063925 15721 raft_consensus.cc:2726] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:40:19.542120 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:19.541597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:45169)
0619 18:40:19.541861 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:19.541972 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260619 18:40:19.674515 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:19.674031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:59379)
0619 18:40:19.674310 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:19.674403 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":85,"threads_started":1}
I20260619 18:40:19.855432 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:19.854968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:47959)
0619 18:40:19.855247 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:19.855302 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260619 18:40:20.543802 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:20.543184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:42093)
0619 18:40:20.543597 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:20.543672 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":108,"threads_started":1}
I20260619 18:40:20.676043 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:20.675475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:56675)
0619 18:40:20.675834 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:20.675911 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
I20260619 18:40:20.856829 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:20.856337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:42911)
0619 18:40:20.856647 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:20.856708 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
I20260619 18:40:21.254173 15728 raft_consensus.cc:482] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:21.254351 15728 raft_consensus.cc:504] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } }
I20260619 18:40:21.254910 15728 leader_election.cc:288] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28bd1f116dca4597bdd411bc83987f53 (127.0.98.3:46683), a8db4a68a7e648648e189299751e2263 (127.0.98.2:45675)
W20260619 18:40:21.255905 15327 leader_election.cc:341] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28bd1f116dca4597bdd411bc83987f53 (127.0.98.3:46683): Not found: Tablet not found: 018c8fe4d76b4a218c0171e2cebfa8c7
W20260619 18:40:21.256035 15327 leader_election.cc:341] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8db4a68a7e648648e189299751e2263 (127.0.98.2:45675): Not found: Tablet not found: 018c8fe4d76b4a218c0171e2cebfa8c7
I20260619 18:40:21.256074 15327 leader_election.cc:302] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fbe0fb5b8c54df5a614d2474e3b9625; no voters: 28bd1f116dca4597bdd411bc83987f53, a8db4a68a7e648648e189299751e2263
I20260619 18:40:21.256211 15728 raft_consensus.cc:2726] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:40:21.545452 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:21.544885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:56251)
0619 18:40:21.545250 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:21.545325 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":148,"threads_started":1}
I20260619 18:40:21.677469 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:21.676957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:59937)
0619 18:40:21.677288 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:21.677355 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":126,"threads_started":1}
I20260619 18:40:21.858146 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:21.857610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:42367)
0619 18:40:21.857964 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:21.858030 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":87,"threads_started":1}
I20260619 18:40:22.546916 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:22.546425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:46531)
0619 18:40:22.546718 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:22.546784 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":97,"threads_started":1}
I20260619 18:40:22.679124 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:22.678538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:57851)
0619 18:40:22.678921 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:22.678997 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":96,"threads_started":1}
I20260619 18:40:22.859549 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:22.858992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:34867)
0619 18:40:22.859364 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:22.859426 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":90,"threads_started":1}
I20260619 18:40:23.548552 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:23.547993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:34321)
0619 18:40:23.548332 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:23.548421 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":123,"threads_started":1}
I20260619 18:40:23.680697 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:23.680177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:58539)
0619 18:40:23.680493 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:23.680573 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":97,"threads_started":1}
I20260619 18:40:23.861078 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:23.860560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:35279)
0619 18:40:23.860857 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:23.860932 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
I20260619 18:40:24.549893 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:24.549376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:54911)
0619 18:40:24.549708 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:24.549772 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":100,"threads_started":1}
I20260619 18:40:24.682157 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:24.681600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:40499)
0619 18:40:24.681958 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:24.682032 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":98,"threads_started":1}
I20260619 18:40:24.862371 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:24.861871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:57785)
0619 18:40:24.862192 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:24.862255 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":93,"threads_started":1}
I20260619 18:40:25.425343 15741 raft_consensus.cc:482] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:25.425510 15741 raft_consensus.cc:504] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } }
I20260619 18:40:25.426043 15741 leader_election.cc:288] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28bd1f116dca4597bdd411bc83987f53 (127.0.98.3:46683), a8db4a68a7e648648e189299751e2263 (127.0.98.2:45675)
W20260619 18:40:25.427012 15327 leader_election.cc:341] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28bd1f116dca4597bdd411bc83987f53 (127.0.98.3:46683): Not found: Tablet not found: 018c8fe4d76b4a218c0171e2cebfa8c7
W20260619 18:40:25.427136 15327 leader_election.cc:341] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8db4a68a7e648648e189299751e2263 (127.0.98.2:45675): Not found: Tablet not found: 018c8fe4d76b4a218c0171e2cebfa8c7
I20260619 18:40:25.427183 15327 leader_election.cc:302] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fbe0fb5b8c54df5a614d2474e3b9625; no voters: 28bd1f116dca4597bdd411bc83987f53, a8db4a68a7e648648e189299751e2263
I20260619 18:40:25.427317 15741 raft_consensus.cc:2726] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:40:25.551249 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:25.550797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:55219)
0619 18:40:25.551061 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:25.551121 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":91,"threads_started":1}
I20260619 18:40:25.683578 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:25.683073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:38689)
0619 18:40:25.683386 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:25.683456 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":93,"threads_started":1}
I20260619 18:40:25.863580 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:25.863135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:45353)
0619 18:40:25.863419 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:25.863481 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":80,"threads_started":1}
I20260619 18:40:26.552438 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:26.551999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:59035)
0619 18:40:26.552278 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:26.552335 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260619 18:40:26.684847 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:26.684397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:36935)
0619 18:40:26.684680 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:26.684742 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
I20260619 18:40:26.864857 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:26.864374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:60831)
0619 18:40:26.864667 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:26.864733 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":90,"threads_started":1}
I20260619 18:40:27.553817 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:27.553278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:52803)
0619 18:40:27.553624 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:27.553694 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":117,"threads_started":1}
I20260619 18:40:27.686348 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:27.685761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:55897)
0619 18:40:27.686132 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:27.686221 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":93,"threads_started":1}
I20260619 18:40:27.866192 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:27.865687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:58281)
0619 18:40:27.866016 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:27.866078 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":103,"threads_started":1}
I20260619 18:40:28.556066 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:28.554668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:56169)
0619 18:40:28.555876 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:28.555949 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1096,"thread_start_us":83,"threads_started":1}
I20260619 18:40:28.687837 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:28.687358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:45399)
0619 18:40:28.687662 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:28.687732 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":92,"threads_started":1}
I20260619 18:40:28.867347 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:28.866922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:60121)
0619 18:40:28.867192 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:28.867241 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":70,"threads_started":1}
I20260619 18:40:29.557528 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:29.557016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:57401)
0619 18:40:29.557328 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:29.557395 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":97,"threads_started":1}
I20260619 18:40:29.689260 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:29.688749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:50265)
0619 18:40:29.689050 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:29.689120 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":93,"threads_started":1}
I20260619 18:40:29.868831 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:29.868277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.3:56021)
0619 18:40:29.868631 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:29.868697 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":136,"threads_started":1}
I20260619 18:40:30.559424 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:30.558850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.1:60299)
0619 18:40:30.559210 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:30.559297 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
I20260619 18:40:30.654232 15758 raft_consensus.cc:482] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:30.654390 15758 raft_consensus.cc:504] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28bd1f116dca4597bdd411bc83987f53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46683 } } peers { permanent_uuid: "a8db4a68a7e648648e189299751e2263" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45675 } } peers { permanent_uuid: "7fbe0fb5b8c54df5a614d2474e3b9625" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42283 } }
I20260619 18:40:30.654836 15758 leader_election.cc:288] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 28bd1f116dca4597bdd411bc83987f53 (127.0.98.3:46683), a8db4a68a7e648648e189299751e2263 (127.0.98.2:45675)
W20260619 18:40:30.655726 15327 leader_election.cc:341] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 28bd1f116dca4597bdd411bc83987f53 (127.0.98.3:46683): Not found: Tablet not found: 018c8fe4d76b4a218c0171e2cebfa8c7
W20260619 18:40:30.655876 15327 leader_election.cc:341] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8db4a68a7e648648e189299751e2263 (127.0.98.2:45675): Not found: Tablet not found: 018c8fe4d76b4a218c0171e2cebfa8c7
I20260619 18:40:30.655932 15327 leader_election.cc:302] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7fbe0fb5b8c54df5a614d2474e3b9625; no voters: 28bd1f116dca4597bdd411bc83987f53, a8db4a68a7e648648e189299751e2263
I20260619 18:40:30.656088 15758 raft_consensus.cc:2726] T 018c8fe4d76b4a218c0171e2cebfa8c7 P 7fbe0fb5b8c54df5a614d2474e3b9625 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:40:30.690923 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:30.690392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38743 (local address 127.0.98.2:34579)
0619 18:40:30.690702 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:30.690771 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":93,"threads_started":1}
I20260619 18:40:30.853011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 15273
I20260619 18:40:30.859504 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 15441
I20260619 18:40:30.865979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 15575
2026-06-19T18:40:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19547 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:40:30 dist-test-slave-rxv6 krb5kdc[15762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:40:30 dist-test-slave-rxv6 krb5kdc[15762](info): set up 2 sockets
Jun 19 18:40:30 dist-test-slave-rxv6 krb5kdc[15762](info): commencing operation
krb5kdc: starting...
W20260619 18:40:31.838227 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:40:31 dist-test-slave-rxv6 krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:40:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:40:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:40:31.957425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34787
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:32.067015 15775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:32.067279 15775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:32.067337 15775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:32.071494 15775 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 18:40:32.071557 15775 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 18:40:32.071573 15775 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 18:40:32.071584 15775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:40:32.071601 15775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:32.071624 15775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:32.071653 15775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:40:32.071671 15775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:40:32.077148 15775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:32.078264 15775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:32.079260 15775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:32.087631 15783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:32.087684 15781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:32.087744 15775 server_base.cc:1034] running on GCE node
W20260619 18:40:32.087638 15780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:32.088392 15775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:32.089499 15775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:32.090641 15775 hybrid_clock.cc:648] HybridClock initialized: now 1781894432090590 us; error 66 us; skew 500 ppm
Jun 19 18:40:32 dist-test-slave-rxv6 krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:32.093335 15775 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:40:32.094262 15775 webserver.cc:460] Webserver started at http://127.0.98.62:40223/ using document root <none> and password file <none>
I20260619 18:40:32.094780 15775 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:32.094875 15775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:32.095090 15775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:32.097060 15775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "a0331d23065d4f039a422c25c02604bd"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "28c1db75494a831209b862452210fd88"
server_key_iv: "40c01d7b2938c50211f85709ff76ca90"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.097568 15775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "a0331d23065d4f039a422c25c02604bd"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "28c1db75494a831209b862452210fd88"
server_key_iv: "40c01d7b2938c50211f85709ff76ca90"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.101110 15775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:40:32.103588 15790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:32.105086 15775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:40:32.105221 15775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "a0331d23065d4f039a422c25c02604bd"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "28c1db75494a831209b862452210fd88"
server_key_iv: "40c01d7b2938c50211f85709ff76ca90"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.105320 15775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:32.121482 15775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:32.124971 15775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:32.125180 15775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:32.131945 15775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34787
I20260619 18:40:32.132179 15842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34787 every 8 connection(s)
I20260619 18:40:32.132918 15775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:40:32.133733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 15775
I20260619 18:40:32.133864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:40:32.134210 392 external_mini_cluster.cc:1428] Setting key 02ebf15f6360a9382392486f083ad7a2
I20260619 18:40:32.136260 15843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:32.142697 15843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Bootstrap starting.
Jun 19 18:40:32 dist-test-slave-rxv6 krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:32.145356 15843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:32.146180 15843 log.cc:824] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:32.148651 15843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: No bootstrap required, opened a new log
I20260619 18:40:32.149444 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.135681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.0.98.62:34787)
0619 18:40:32.136073 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:32.136089 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:32.136125 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:40:32.136807 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:32.136825 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:32.137159 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:32.137529 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:32.138470 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.139515 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.140643 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.141163 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.141286 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.144344 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:32.144350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:32.144365 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:32.144405 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:32.146334 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:32.146862 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:32.146869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:32.146876 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:32.147028 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:32.147346 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:32.147348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:32.147349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:32.147857 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:32.148021 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:32.148333 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 18:40:32.148628 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":112,"threads_started":1}
I20260619 18:40:32.152654 15843 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } }
I20260619 18:40:32.152890 15843 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:32.152945 15843 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0331d23065d4f039a422c25c02604bd, State: Initialized, Role: FOLLOWER
I20260619 18:40:32.153340 15843 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } }
I20260619 18:40:32.153461 15843 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:40:32.153522 15843 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:40:32.153611 15843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:32.154542 15843 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } }
I20260619 18:40:32.154882 15843 leader_election.cc:302] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0331d23065d4f039a422c25c02604bd; no voters:
I20260619 18:40:32.155192 15843 leader_election.cc:288] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:40:32.155333 15848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:32.155562 15848 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 1 LEADER]: Becoming Leader. State: Replica: a0331d23065d4f039a422c25c02604bd, State: Running, Role: LEADER
I20260619 18:40:32.155910 15848 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } }
I20260619 18:40:32.156360 15843 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:40:32.157459 15849 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0331d23065d4f039a422c25c02604bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } } }
I20260619 18:40:32.157580 15849 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:32.157976 15850 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0331d23065d4f039a422c25c02604bd. Latest consensus state: current_term: 1 leader_uuid: "a0331d23065d4f039a422c25c02604bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } } }
I20260619 18:40:32.158638 15850 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [sys.catalog]: This master's current role is: LEADER
W20260619 18:40:32.161258 15863 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:40:32.161321 15863 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:40:32.161375 15857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:40:32.162074 15857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:40:32.167490 15857 catalog_manager.cc:1347] Generated new cluster ID: 824cfb8382de4a79a7f5fe465e7dc89f
I20260619 18:40:32.167557 15857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:40:32.177246 15857 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:40:32.178517 15857 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:40:32.182989 15857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Generated new TSK 0
I20260619 18:40:32.183753 15857 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:40:32.225337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34787
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:32.352869 15869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:32.353147 15869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:32.353207 15869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:32.357467 15869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:32.357540 15869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:32.357630 15869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:40:32.363236 15869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:32.364377 15869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:32.365417 15869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:32.375234 15877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:32.375240 15874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:32.375228 15875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:32.375442 15869 server_base.cc:1034] running on GCE node
I20260619 18:40:32.375986 15869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:32.376658 15869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:32.377882 15869 hybrid_clock.cc:648] HybridClock initialized: now 1781894432377868 us; error 30 us; skew 500 ppm
Jun 19 18:40:32 dist-test-slave-rxv6 krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:32.380713 15869 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:40:32.381624 15869 webserver.cc:460] Webserver started at http://127.0.98.1:44863/ using document root <none> and password file <none>
I20260619 18:40:32.382236 15869 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:32.382297 15869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:32.382468 15869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:32.384539 15869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "77e1fea46ca94640bca652336c0a1f84"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "d06453d28f4a93803d70cd08448f8de7"
server_key_iv: "3b5fc911a7571303f8382efe67974f64"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.385061 15869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "77e1fea46ca94640bca652336c0a1f84"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "d06453d28f4a93803d70cd08448f8de7"
server_key_iv: "3b5fc911a7571303f8382efe67974f64"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.388908 15869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:40:32.391474 15884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:32.392817 15869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:40:32.392935 15869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "77e1fea46ca94640bca652336c0a1f84"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "d06453d28f4a93803d70cd08448f8de7"
server_key_iv: "3b5fc911a7571303f8382efe67974f64"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.393041 15869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:32.409792 15869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:32.412817 15869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:32.413023 15869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:32.413614 15869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:32.414466 15869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:32.414515 15869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:32.414567 15869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:32.414597 15869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:32.423377 15869 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37365
I20260619 18:40:32.423449 15997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37365 every 8 connection(s)
I20260619 18:40:32.424311 15869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:40:32.432658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 15869
I20260619 18:40:32.432811 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:40:32.433142 392 external_mini_cluster.cc:1428] Setting key fa4e79f8a560b9aa175ae7226ea5a7cd
Jun 19 18:40:32 dist-test-slave-rxv6 krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:32.438557 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.426175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53541 (local address 127.0.98.62:34787)
0619 18:40:32.426385 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:32.426389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:32.427489 (+ 1100us) server_negotiation.cc:407] Connection header received
0619 18:40:32.428443 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:32.428446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:32.428506 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:32.428581 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:32.429808 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.430406 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.431584 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.431818 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.431906 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.434952 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:32.434955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:32.434957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:32.434982 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:32.436359 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:32.436907 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:32.436911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:32.436912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:32.436999 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:32.437343 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:32.437346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:32.437348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:32.437546 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:32.437634 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:32.438241 (+ 607us) server_negotiation.cc:299] Negotiation successful
0619 18:40:32.438388 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 18:40:32.439431 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.426649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:53541)
0619 18:40:32.427324 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:32.427366 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:32.428224 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:32.428767 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:32.428782 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:32.429153 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:32.429646 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.429660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.430534 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.430536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.431445 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.431455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.431961 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.431965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.432130 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.432720 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:32.432741 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:32.434787 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:32.436490 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:32.436497 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:32.436515 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:32.436806 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:32.437085 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:32.437088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:32.437090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:32.437250 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:32.437648 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:32.437656 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:32.437980 (+ 324us) client_negotiation.cc:769] Sending connection context
0619 18:40:32.438185 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 18:40:32.438541 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":114,"threads_started":1}
I20260619 18:40:32.441030 15998 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34787
I20260619 18:40:32.441299 15998 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:32.441838 15998 heartbeater.cc:511] Master 127.0.98.62:34787 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 18:40:32.443552 15807 ts_manager.cc:194] Registered new tserver with Master: 77e1fea46ca94640bca652336c0a1f84 (127.0.98.1:37365)
I20260619 18:40:32.444800 15807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:40:32.461062 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34787
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:32.568548 16003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:32.568802 16003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:32.568846 16003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:32.572963 16003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:32.573027 16003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:32.573086 16003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:40:32.578369 16003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:32.579372 16003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:32.580292 16003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:32.589475 16011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:32.589496 16008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:32.589519 16009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:32.589890 16003 server_base.cc:1034] running on GCE node
I20260619 18:40:32.590291 16003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:32.590935 16003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:32.592101 16003 hybrid_clock.cc:648] HybridClock initialized: now 1781894432592082 us; error 35 us; skew 500 ppm
Jun 19 18:40:32 dist-test-slave-rxv6 krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:32.594848 16003 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:40:32.595697 16003 webserver.cc:460] Webserver started at http://127.0.98.2:35335/ using document root <none> and password file <none>
I20260619 18:40:32.596233 16003 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:32.596303 16003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:32.596535 16003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:32.598721 16003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "6632af5cdb994949abff101dc7f9c38c"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "30e26f921216a26be57ca43b0accba38"
server_key_iv: "fa7247a32ca3350617b8f24145354e7d"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.599282 16003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6632af5cdb994949abff101dc7f9c38c"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "30e26f921216a26be57ca43b0accba38"
server_key_iv: "fa7247a32ca3350617b8f24145354e7d"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.603209 16003 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:40:32.605640 16018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:32.606936 16003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:40:32.607033 16003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "6632af5cdb994949abff101dc7f9c38c"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "30e26f921216a26be57ca43b0accba38"
server_key_iv: "fa7247a32ca3350617b8f24145354e7d"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.607144 16003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:32.623508 16003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:32.626945 16003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:32.627116 16003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:32.627710 16003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:32.628525 16003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:32.628572 16003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:32.628618 16003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:32.628646 16003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:32.637176 16003 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34287
I20260619 18:40:32.637238 16131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34287 every 8 connection(s)
I20260619 18:40:32.638177 16003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 18:40:32 dist-test-slave-rxv6 krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:32.647555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 16003
I20260619 18:40:32.647666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:40:32.647938 392 external_mini_cluster.cc:1428] Setting key 1ac845b8383c8841cf568e1120e69012
I20260619 18:40:32.651422 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.640118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41081 (local address 127.0.98.62:34787)
0619 18:40:32.640257 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:32.640261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:32.641221 (+ 960us) server_negotiation.cc:407] Connection header received
0619 18:40:32.642128 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:32.642131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:32.642178 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:32.642244 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:32.643335 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.643876 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.645051 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.645267 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.645347 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.647812 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:32.647815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:32.647818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:32.647847 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:32.649309 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:32.649803 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:32.649806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:32.649808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:32.649899 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:32.650225 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:32.650228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:32.650230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:32.650436 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:32.650527 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:32.651191 (+ 664us) server_negotiation.cc:299] Negotiation successful
0619 18:40:32.651302 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:40:32.652490 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.640465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:41081)
0619 18:40:32.641099 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:32.641139 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:32.641897 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:32.642408 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:32.642414 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:32.642747 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:32.643206 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.643219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.643970 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.643972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.644946 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.644956 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.645397 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.645401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.645586 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.646147 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:32.646163 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:32.647633 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:32.649417 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:32.649422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:32.649434 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:32.649708 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:32.650001 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:32.650004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:32.650005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:32.650151 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:32.650570 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:32.650575 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:32.650878 (+ 303us) client_negotiation.cc:769] Sending connection context
0619 18:40:32.651114 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 18:40:32.651474 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":113,"threads_started":1}
I20260619 18:40:32.653656 16132 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34787
I20260619 18:40:32.653882 16132 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:32.654379 16132 heartbeater.cc:511] Master 127.0.98.62:34787 requested a full tablet report, sending...
I20260619 18:40:32.655441 15807 ts_manager.cc:194] Registered new tserver with Master: 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287)
I20260619 18:40:32.655992 15807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41081
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:40:32.675055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34787
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:32.784552 16137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:32.784823 16137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:32.784881 16137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:32.788908 16137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:32.788967 16137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:32.789047 16137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:40:32.795080 16137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:32.796545 16137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:32.797945 16137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:32.810091 16145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:32.810073 16143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:32.810073 16142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:32.810514 16137 server_base.cc:1034] running on GCE node
I20260619 18:40:32.810969 16137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:32.811748 16137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:32.812868 16137 hybrid_clock.cc:648] HybridClock initialized: now 1781894432812847 us; error 38 us; skew 500 ppm
Jun 19 18:40:32 dist-test-slave-rxv6 krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:32.816105 16137 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:40:32.817165 16137 webserver.cc:460] Webserver started at http://127.0.98.3:39059/ using document root <none> and password file <none>
I20260619 18:40:32.817751 16137 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:32.817845 16137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:32.818122 16137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:40:32.820366 16137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "aad822701eb844d18769f67d29d5a67a"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "682ce07c64c353cbb4e1ee87a6f6e887"
server_key_iv: "0e223dff2e5c894490922c218d5ddac9"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.820889 16137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aad822701eb844d18769f67d29d5a67a"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "682ce07c64c353cbb4e1ee87a6f6e887"
server_key_iv: "0e223dff2e5c894490922c218d5ddac9"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.824654 16137 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:40:32.827041 16152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:32.828343 16137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:40:32.828438 16137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "aad822701eb844d18769f67d29d5a67a"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "682ce07c64c353cbb4e1ee87a6f6e887"
server_key_iv: "0e223dff2e5c894490922c218d5ddac9"
server_key_version: "encryptionkey@0"
I20260619 18:40:32.828524 16137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:32.840029 16137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:32.842998 16137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:32.843150 16137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:32.843713 16137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:32.844508 16137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:32.844559 16137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:32.844597 16137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:32.844619 16137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:32.852908 16137 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39585
I20260619 18:40:32.852938 16265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39585 every 8 connection(s)
I20260619 18:40:32.853790 16137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:40:32.861173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 16137
I20260619 18:40:32.861292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:40:32.861603 392 external_mini_cluster.cc:1428] Setting key 4206ca564ee979e19ecbc4ad8cdcc2ad
Jun 19 18:40:32 dist-test-slave-rxv6 krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:32.866452 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.855523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60563 (local address 127.0.98.62:34787)
0619 18:40:32.855644 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:32.855647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:32.856623 (+ 976us) server_negotiation.cc:407] Connection header received
0619 18:40:32.857517 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:32.857521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:32.857580 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:32.857653 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:32.858829 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.859408 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.860498 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.860712 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.860797 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.863351 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:32.863354 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:32.863356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:32.863384 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:32.864679 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:32.865144 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:32.865146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:32.865147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:32.865211 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:32.865457 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:32.865459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:32.865460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:32.865631 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:32.865699 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:32.866239 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 18:40:32.866337 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:40:32.867254 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.855850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:60563)
0619 18:40:32.856480 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:32.856520 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:32.857320 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:32.857820 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:32.857826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:32.858222 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:32.858689 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.858703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.859524 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.859527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.860380 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.860389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.860845 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.860849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.861028 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.861664 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:32.861685 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:32.863207 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:32.864799 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:32.864805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:32.864819 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:32.865068 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:32.865271 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:32.865273 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:32.865274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:32.865397 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:32.865736 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:32.865741 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:32.866041 (+ 300us) client_negotiation.cc:769] Sending connection context
0619 18:40:32.866209 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 18:40:32.866473 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":110,"threads_started":1}
I20260619 18:40:32.868402 16266 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34787
I20260619 18:40:32.868603 16266 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:32.869058 16266 heartbeater.cc:511] Master 127.0.98.62:34787 requested a full tablet report, sending...
I20260619 18:40:32.870052 15807 ts_manager.cc:194] Registered new tserver with Master: aad822701eb844d18769f67d29d5a67a (127.0.98.3:39585)
I20260619 18:40:32.870604 15807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60563
I20260619 18:40:32.875200 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:40:32.884065 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.877196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.0.98.62:34787)
0619 18:40:32.877344 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:32.877349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:32.877459 (+ 110us) server_negotiation.cc:407] Connection header received
0619 18:40:32.877553 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:32.877555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:32.877591 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:32.877657 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:32.878398 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.878953 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.879729 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.879973 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.880049 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.881276 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:32.881279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:32.881281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:32.881303 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:32.882627 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:32.883009 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:32.883012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:32.883013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:32.883082 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:32.883298 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:32.883300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:32.883301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:32.883526 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:32.883647 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:32.883848 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 18:40:32.883943 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:40:32.887090 15807 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:40:32.890064 15807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:40:32.904350 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.899661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.0.98.2:34287)
0619 18:40:32.899940 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:32.899952 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:32.899974 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:40:32.900048 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:32.900054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:32.900263 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:32.900426 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:32.900732 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.901591 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.903115 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.903659 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.903806 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.903964 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:32.904111 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 18:40:32.904203 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":93,"threads_started":1}
I20260619 18:40:32.904349 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.898517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34287 (local address 127.0.0.1:51860)
0619 18:40:32.898810 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:32.898848 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:32.898957 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:32.900426 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:32.900432 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:32.900456 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:32.900606 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.900613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.901831 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.901834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.902912 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.902921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.903815 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.903818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.903921 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.903961 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 18:40:32.904078 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 18:40:32.904188 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":1,"thread_start_us":67,"threads_started":1}
I20260619 18:40:32.905048 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.898589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37365 (local address 127.0.0.1:38128)
0619 18:40:32.899430 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:32.899445 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:32.899535 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:32.900671 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:32.900674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:32.900691 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:32.900854 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.900863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.902349 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.902353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.903621 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.903632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.904693 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.904696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.904777 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.904804 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 18:40:32.904853 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 18:40:32.904921 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":9,"thread_start_us":89,"threads_started":1}
I20260619 18:40:32.905239 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.899370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56244 (local address 127.0.98.3:39585)
0619 18:40:32.899798 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:32.899806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:32.900505 (+ 699us) server_negotiation.cc:407] Connection header received
0619 18:40:32.900584 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:32.900589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:32.900765 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:32.900919 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:32.901098 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.902226 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.903753 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.904538 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.904629 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.904751 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:32.905022 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 18:40:32.905113 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":128,"threads_started":1}
I20260619 18:40:32.905220 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.899198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39585 (local address 127.0.0.1:56244)
0619 18:40:32.900327 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:32.900339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:32.900420 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:32.900901 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:32.900904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:32.900920 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:32.901005 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.901012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.902362 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.902364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.903624 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.903632 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.904811 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.904814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.904893 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.904912 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 18:40:32.904978 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 18:40:32.905048 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1056,"mutex_wait_us":106,"thread_start_us":111,"threads_started":1}
I20260619 18:40:32.905958 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.900002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38128 (local address 127.0.98.1:37365)
0619 18:40:32.900240 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:32.900251 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:32.900269 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:40:32.900326 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:32.900330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:32.900532 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:32.900695 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:32.900984 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.902192 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.903792 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.904568 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.905514 (+ 946us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.905647 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:32.905696 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 18:40:32.905797 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":73,"threads_started":1}
I20260619 18:40:32.907773 16066 tablet_service.cc:1467] Processing CreateTablet for tablet 54c94f945e314548b5c2f51eb2366848 (DEFAULT_TABLE table=test-table [id=4f559636b3c3407e8ebe4747fca23f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:32.908002 16200 tablet_service.cc:1467] Processing CreateTablet for tablet 54c94f945e314548b5c2f51eb2366848 (DEFAULT_TABLE table=test-table [id=4f559636b3c3407e8ebe4747fca23f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:32.908923 16066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c94f945e314548b5c2f51eb2366848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:32.908938 16200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c94f945e314548b5c2f51eb2366848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:32.908622 15932 tablet_service.cc:1467] Processing CreateTablet for tablet 54c94f945e314548b5c2f51eb2366848 (DEFAULT_TABLE table=test-table [id=4f559636b3c3407e8ebe4747fca23f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:32.909476 15932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54c94f945e314548b5c2f51eb2366848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:32.915758 16284 tablet_bootstrap.cc:492] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: Bootstrap starting.
I20260619 18:40:32.916303 16283 tablet_bootstrap.cc:492] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a: Bootstrap starting.
I20260619 18:40:32.916750 16285 tablet_bootstrap.cc:492] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84: Bootstrap starting.
I20260619 18:40:32.918150 16284 tablet_bootstrap.cc:654] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:32.918382 16283 tablet_bootstrap.cc:654] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:32.918712 16285 tablet_bootstrap.cc:654] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:32.918902 16284 log.cc:824] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:32.919627 16283 log.cc:824] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:32.919955 16285 log.cc:824] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:32.921029 16284 tablet_bootstrap.cc:492] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: No bootstrap required, opened a new log
I20260619 18:40:32.921216 16284 ts_tablet_manager.cc:1397] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260619 18:40:32.921607 16283 tablet_bootstrap.cc:492] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a: No bootstrap required, opened a new log
I20260619 18:40:32.921679 16285 tablet_bootstrap.cc:492] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84: No bootstrap required, opened a new log
I20260619 18:40:32.921792 16283 ts_tablet_manager.cc:1397] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:40:32.921813 16285 ts_tablet_manager.cc:1397] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 18:40:32.924150 16284 raft_consensus.cc:348] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } }
I20260619 18:40:32.924393 16284 raft_consensus.cc:374] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:32.924427 16284 raft_consensus.cc:729] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6632af5cdb994949abff101dc7f9c38c, State: Initialized, Role: FOLLOWER
I20260619 18:40:32.924840 16284 consensus_queue.cc:260] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } }
I20260619 18:40:32.925463 16132 heartbeater.cc:503] Master 127.0.98.62:34787 was elected leader, sending a full tablet report...
I20260619 18:40:32.925560 16285 raft_consensus.cc:348] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } }
I20260619 18:40:32.925560 16283 raft_consensus.cc:348] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } }
I20260619 18:40:32.925807 16283 raft_consensus.cc:374] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:32.925806 16285 raft_consensus.cc:374] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:32.925861 16285 raft_consensus.cc:729] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e1fea46ca94640bca652336c0a1f84, State: Initialized, Role: FOLLOWER
I20260619 18:40:32.925933 16283 raft_consensus.cc:729] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad822701eb844d18769f67d29d5a67a, State: Initialized, Role: FOLLOWER
I20260619 18:40:32.926319 16283 consensus_queue.cc:260] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } }
I20260619 18:40:32.926319 16285 consensus_queue.cc:260] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } }
I20260619 18:40:32.926955 16284 ts_tablet_manager.cc:1428] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260619 18:40:32.927273 16285 ts_tablet_manager.cc:1428] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260619 18:40:32.927273 16283 ts_tablet_manager.cc:1428] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 18:40:32.927618 16266 heartbeater.cc:503] Master 127.0.98.62:34787 was elected leader, sending a full tablet report...
I20260619 18:40:32.928560 15998 heartbeater.cc:503] Master 127.0.98.62:34787 was elected leader, sending a full tablet report...
I20260619 18:40:32.943884 16291 raft_consensus.cc:482] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:32.944108 16291 raft_consensus.cc:504] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } }
I20260619 18:40:32.945319 16291 leader_election.cc:288] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287), 77e1fea46ca94640bca652336c0a1f84 (127.0.98.1:37365)
I20260619 18:40:32.948833 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.945543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34287 (local address 127.0.98.3:35031)
0619 18:40:32.945724 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:32.945739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:32.945834 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:32.946084 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:32.946086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:32.946102 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:32.946213 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.946218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.946897 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.946901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.947693 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.947703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.948443 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.948447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.948548 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.948569 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:40:32.948630 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 18:40:32.948698 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":1}
I20260619 18:40:32.948834 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.945646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35031 (local address 127.0.98.2:34287)
0619 18:40:32.945780 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:32.945784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:32.945799 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:40:32.945961 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:32.945964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:32.946005 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:32.946069 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:32.946313 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.946796 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.947837 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.948347 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.948445 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.948508 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:32.948656 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 18:40:32.948719 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 18:40:32.949085 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.945702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42233 (local address 127.0.98.1:37365)
0619 18:40:32.945819 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:32.945823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:32.946078 (+ 255us) server_negotiation.cc:407] Connection header received
0619 18:40:32.946190 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:32.946193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:32.946233 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:32.946294 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:32.946541 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.947007 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.948142 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.948613 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.948703 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.948757 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:32.948912 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 18:40:32.948973 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:40:32.949105 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.945575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37365 (local address 127.0.98.3:42233)
0619 18:40:32.945971 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:32.945982 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:32.946099 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:32.946312 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:32.946314 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:32.946328 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:32.946433 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.946438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.947102 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.947105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.948033 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:32.948042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.948716 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.948719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:32.948809 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.948830 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:40:32.948892 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 18:40:32.948954 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":117,"threads_started":1}
I20260619 18:40:32.949536 16086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c94f945e314548b5c2f51eb2366848" candidate_uuid: "aad822701eb844d18769f67d29d5a67a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6632af5cdb994949abff101dc7f9c38c" is_pre_election: true
I20260619 18:40:32.949602 15952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c94f945e314548b5c2f51eb2366848" candidate_uuid: "aad822701eb844d18769f67d29d5a67a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e1fea46ca94640bca652336c0a1f84" is_pre_election: true
I20260619 18:40:32.949820 16086 raft_consensus.cc:2454] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad822701eb844d18769f67d29d5a67a in term 0.
I20260619 18:40:32.949867 15952 raft_consensus.cc:2454] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aad822701eb844d18769f67d29d5a67a in term 0.
I20260619 18:40:32.950290 16155 leader_election.cc:302] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6632af5cdb994949abff101dc7f9c38c, aad822701eb844d18769f67d29d5a67a; no voters:
I20260619 18:40:32.950536 16291 raft_consensus.cc:2781] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:40:32.950615 16291 raft_consensus.cc:482] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:40:32.950661 16291 raft_consensus.cc:3037] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:32.951617 16291 raft_consensus.cc:504] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } }
I20260619 18:40:32.951928 16291 leader_election.cc:288] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [CANDIDATE]: Term 1 election: Requested vote from peers 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287), 77e1fea46ca94640bca652336c0a1f84 (127.0.98.1:37365)
I20260619 18:40:32.952230 16086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c94f945e314548b5c2f51eb2366848" candidate_uuid: "aad822701eb844d18769f67d29d5a67a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6632af5cdb994949abff101dc7f9c38c"
I20260619 18:40:32.952281 15952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54c94f945e314548b5c2f51eb2366848" candidate_uuid: "aad822701eb844d18769f67d29d5a67a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e1fea46ca94640bca652336c0a1f84"
I20260619 18:40:32.952335 16086 raft_consensus.cc:3037] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:32.952376 15952 raft_consensus.cc:3037] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:32.953303 16086 raft_consensus.cc:2454] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad822701eb844d18769f67d29d5a67a in term 1.
I20260619 18:40:32.953359 15952 raft_consensus.cc:2454] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aad822701eb844d18769f67d29d5a67a in term 1.
I20260619 18:40:32.953564 16155 leader_election.cc:302] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6632af5cdb994949abff101dc7f9c38c, aad822701eb844d18769f67d29d5a67a; no voters:
I20260619 18:40:32.953722 16291 raft_consensus.cc:2781] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:32.953946 16291 raft_consensus.cc:686] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [term 1 LEADER]: Becoming Leader. State: Replica: aad822701eb844d18769f67d29d5a67a, State: Running, Role: LEADER
I20260619 18:40:32.954274 16291 consensus_queue.cc:237] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } }
I20260619 18:40:32.957132 15807 catalog_manager.cc:5495] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a reported cstate change: term changed from 0 to 1, leader changed from <none> to aad822701eb844d18769f67d29d5a67a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "aad822701eb844d18769f67d29d5a67a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } health_report { overall_health: HEALTHY } } }
I20260619 18:40:32.968626 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:32.965251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56252 (local address 127.0.98.3:39585)
0619 18:40:32.965393 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:32.965397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:32.965410 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:40:32.965481 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:32.965483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:32.965533 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:32.965624 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:32.965858 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.966425 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.967259 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:32.967486 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:32.967570 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:32.967849 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:32.968252 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:32.968363 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:32.968423 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 18:40:32.968477 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:40:32.974344 15952 raft_consensus.cc:1261] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Refusing update from remote peer aad822701eb844d18769f67d29d5a67a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:32.974370 16086 raft_consensus.cc:1261] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c [term 1 FOLLOWER]: Refusing update from remote peer aad822701eb844d18769f67d29d5a67a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:32.975078 16293 consensus_queue.cc:1035] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [LEADER]: Connected to new peer: Peer: permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:32.975255 16291 consensus_queue.cc:1035] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:32.981943 16298 mvcc.cc:205] Tried to move back new op lower bound from 7298639597455581184 to 7298639597383233536. Current Snapshot: MvccSnapshot[applied={T|T < 7298639597455581184}]
I20260619 18:40:32.983438 16297 mvcc.cc:205] Tried to move back new op lower bound from 7298639597455581184 to 7298639597383233536. Current Snapshot: MvccSnapshot[applied={T|T < 7298639597455581184}]
I20260619 18:40:32.990082 15805 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:32.990232 15805 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:32.992352 15805 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Sending DeleteTablet for 3 replicas of tablet 54c94f945e314548b5c2f51eb2366848
I20260619 18:40:32.993108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 15869
I20260619 18:40:32.993124 16066 tablet_service.cc:1514] Processing DeleteTablet for tablet 54c94f945e314548b5c2f51eb2366848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51860
I20260619 18:40:32.993665 16200 tablet_service.cc:1514] Processing DeleteTablet for tablet 54c94f945e314548b5c2f51eb2366848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56244
I20260619 18:40:32.998188 16306 tablet_replica.cc:330] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: stopping tablet replica
I20260619 18:40:32.998721 16306 raft_consensus.cc:2229] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:32.998946 16307 tablet_replica.cc:330] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a: stopping tablet replica
I20260619 18:40:32.999174 16306 raft_consensus.cc:2258] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 18:40:32.999248 15791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37365 (error 108)
I20260619 18:40:32.999265 16307 raft_consensus.cc:2229] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:40:32.999632 16307 raft_consensus.cc:2258] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:33.000715 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:33.000372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37365 (local address 127.0.0.1:38142)
0619 18:40:33.000566 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:33.000632 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260619 18:40:33.000778 16306 ts_tablet_manager.cc:1905] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 18:40:33.000983 15791 catalog_manager.cc:4584] TS 77e1fea46ca94640bca652336c0a1f84 (127.0.98.1:37365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 54c94f945e314548b5c2f51eb2366848: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37365: connect: Connection refused (error 111)
I20260619 18:40:33.001041 16307 ts_tablet_manager.cc:1905] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:33.001367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 16003
I20260619 18:40:33.003655 16307 ts_tablet_manager.cc:1918] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:40:33.003741 16307 log.cc:1196] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/54c94f945e314548b5c2f51eb2366848
I20260619 18:40:33.004065 16307 ts_tablet_manager.cc:1939] T 54c94f945e314548b5c2f51eb2366848 P aad822701eb844d18769f67d29d5a67a: Deleting consensus metadata
I20260619 18:40:33.005827 15791 catalog_manager.cc:4841] TS aad822701eb844d18769f67d29d5a67a (127.0.98.3:39585): tablet 54c94f945e314548b5c2f51eb2366848 (table test-table [id=4f559636b3c3407e8ebe4747fca23f97]) successfully deleted
I20260619 18:40:33.007268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 16137
I20260619 18:40:33.007498 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:33.007253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34287 (local address 127.0.0.1:51864)
0619 18:40:33.007376 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:33.007426 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 18:40:33.012346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 15775
I20260619 18:40:53.017714 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34787
--webserver_interface=127.0.98.62
--webserver_port=40223
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:40:53.145128 16309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:53.145462 16309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:53.145519 16309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:53.149730 16309 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 18:40:53.149802 16309 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 18:40:53.149823 16309 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 18:40:53.149842 16309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:40:53.149863 16309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:53.149888 16309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:53.149945 16309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:40:53.149978 16309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:40:53.155635 16309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:53.156965 16309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:53.158275 16309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:53.173180 16315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:53.173485 16316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:53.173377 16309 server_base.cc:1034] running on GCE node
W20260619 18:40:53.173180 16318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:53.174332 16309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:53.175766 16309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:53.176934 16309 hybrid_clock.cc:648] HybridClock initialized: now 1781894453176902 us; error 44 us; skew 500 ppm
Jun 19 18:40:53 dist-test-slave-rxv6 krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:53.183804 16309 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:40:53.184913 16309 webserver.cc:460] Webserver started at http://127.0.98.62:40223/ using document root <none> and password file <none>
I20260619 18:40:53.185586 16309 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:53.185665 16309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:53.190450 16309 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:40:53.192940 16325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:53.194499 16309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:40:53.194624 16309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "a0331d23065d4f039a422c25c02604bd"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "28c1db75494a831209b862452210fd88"
server_key_iv: "40c01d7b2938c50211f85709ff76ca90"
server_key_version: "encryptionkey@0"
I20260619 18:40:53.195330 16309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:53.230999 16309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:53.234935 16309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:53.235149 16309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:53.243559 16309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34787
I20260619 18:40:53.243888 16377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34787 every 8 connection(s)
I20260619 18:40:53.244946 16309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:40:53.246155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 16309
I20260619 18:40:53.246650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37365
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34787
--webserver_port=44863
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:40:53.251075 16378 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:40:53.253782 16378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Bootstrap starting.
I20260619 18:40:53.279508 16378 log.cc:824] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:53.291286 16378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:40:53.291683 16378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Bootstrap complete.
I20260619 18:40:53.296168 16378 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } }
I20260619 18:40:53.296617 16378 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0331d23065d4f039a422c25c02604bd, State: Initialized, Role: FOLLOWER
I20260619 18:40:53.297184 16378 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } }
I20260619 18:40:53.297318 16378 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:40:53.297392 16378 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:40:53.297490 16378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 1 FOLLOWER]: Advancing to term 2
I20260619 18:40:53.299748 16378 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } }
I20260619 18:40:53.300110 16378 leader_election.cc:302] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0331d23065d4f039a422c25c02604bd; no voters:
I20260619 18:40:53.300457 16378 leader_election.cc:288] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 18:40:53.300582 16382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 2 FOLLOWER]: Leader election won for term 2
I20260619 18:40:53.300853 16382 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 2 LEADER]: Becoming Leader. State: Replica: a0331d23065d4f039a422c25c02604bd, State: Running, Role: LEADER
I20260619 18:40:53.301234 16382 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } }
I20260619 18:40:53.301836 16378 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:40:53.302426 16382 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0331d23065d4f039a422c25c02604bd. Latest consensus state: current_term: 2 leader_uuid: "a0331d23065d4f039a422c25c02604bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } } }
I20260619 18:40:53.302546 16382 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:53.302939 16387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:40:53.303051 16383 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a0331d23065d4f039a422c25c02604bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } } }
I20260619 18:40:53.303241 16383 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [sys.catalog]: This master's current role is: LEADER
I20260619 18:40:53.308507 16387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:40:53.309293 16387 catalog_manager.cc:1259] Loaded cluster ID: 824cfb8382de4a79a7f5fe465e7dc89f
I20260619 18:40:53.309369 16387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:40:53.312060 16387 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:40:53.312800 16387 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Loaded TSK: 0
I20260619 18:40:53.336272 16387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Generated new TSK 1
I20260619 18:40:53.336663 16387 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:40:53.397279 16380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:53.397564 16380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:53.397610 16380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:53.401813 16380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:53.401882 16380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:53.401990 16380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:40:53.407819 16380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44863
--tserver_master_addrs=127.0.98.62:34787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:53.409153 16380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:53.410404 16380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:53.421063 16406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:53.421214 16380 server_base.cc:1034] running on GCE node
W20260619 18:40:53.421068 16408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:53.421386 16405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:53.421808 16380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:53.422562 16380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:53.423744 16380 hybrid_clock.cc:648] HybridClock initialized: now 1781894453423730 us; error 29 us; skew 500 ppm
Jun 19 18:40:53 dist-test-slave-rxv6 krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:53.426659 16380 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:40:53.427682 16380 webserver.cc:460] Webserver started at http://127.0.98.1:44863/ using document root <none> and password file <none>
I20260619 18:40:53.428319 16380 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:53.428397 16380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:53.432883 16380 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:40:53.435452 16415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:53.436908 16380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 18:40:53.437024 16380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "77e1fea46ca94640bca652336c0a1f84"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "d06453d28f4a93803d70cd08448f8de7"
server_key_iv: "3b5fc911a7571303f8382efe67974f64"
server_key_version: "encryptionkey@0"
I20260619 18:40:53.437671 16380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:53.456637 16380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:53.460347 16380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:53.460552 16380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:53.461253 16380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:53.462546 16422 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 18:40:53.467626 16380 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 18:40:53.467705 16380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 18:40:53.467764 16380 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 18:40:53.469707 16380 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 18:40:53.469750 16380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 18:40:53.470108 16422 tablet_bootstrap.cc:492] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84: Bootstrap starting.
I20260619 18:40:53.479269 16380 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37365
I20260619 18:40:53.479337 16529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37365 every 8 connection(s)
I20260619 18:40:53.480424 16380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:40:53.485363 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 16380
I20260619 18:40:53.485883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34287
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34787
--webserver_port=35335
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:40:53.490618 16422 log.cc:824] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:40:53 dist-test-slave-rxv6 krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:53.497843 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:53.482686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:33835)
0619 18:40:53.483228 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:53.483272 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:53.484114 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:53.485233 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:53.485242 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:53.485659 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:53.486191 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:53.486205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.488001 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.488005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:53.489416 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:53.489428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.490306 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.490310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:53.490499 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:53.491099 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:53.491120 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:53.492752 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:53.494744 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:53.494751 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:53.494765 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:53.495018 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:53.495356 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:53.495359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:53.495362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:53.495511 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:53.496087 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:53.496094 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:53.496328 (+ 234us) client_negotiation.cc:769] Sending connection context
0619 18:40:53.496505 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 18:40:53.496862 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":141,"threads_started":1}
I20260619 18:40:53.497841 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:53.482649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33835 (local address 127.0.98.62:34787)
0619 18:40:53.483033 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:53.483055 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:53.483402 (+ 347us) server_negotiation.cc:407] Connection header received
0619 18:40:53.484542 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:53.484553 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:53.484854 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:53.485161 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:53.486360 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.487793 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.489570 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.490138 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.490342 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:53.492952 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:53.492957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:53.492970 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:53.492999 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:53.494622 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:53.495131 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:53.495136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:53.495140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:53.495266 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:53.495600 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:53.495603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:53.495604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:53.495975 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:53.496098 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:53.496640 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 18:40:53.496903 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":118,"threads_started":1}
I20260619 18:40:53.500494 16530 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34787
I20260619 18:40:53.500698 16422 tablet_bootstrap.cc:492] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:40:53.500772 16530 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:53.501112 16422 tablet_bootstrap.cc:492] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84: Bootstrap complete.
I20260619 18:40:53.501329 16530 heartbeater.cc:511] Master 127.0.98.62:34787 requested a full tablet report, sending...
I20260619 18:40:53.502521 16422 ts_tablet_manager.cc:1397] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.007s
I20260619 18:40:53.504024 16342 ts_manager.cc:194] Registered new tserver with Master: 77e1fea46ca94640bca652336c0a1f84 (127.0.98.1:37365)
I20260619 18:40:53.505659 16422 raft_consensus.cc:348] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } }
I20260619 18:40:53.506258 16422 raft_consensus.cc:729] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e1fea46ca94640bca652336c0a1f84, State: Initialized, Role: FOLLOWER
I20260619 18:40:53.506858 16422 consensus_queue.cc:260] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } }
I20260619 18:40:53.507809 16342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33835
I20260619 18:40:53.507956 16422 ts_tablet_manager.cc:1428] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 18:40:53.510294 16530 heartbeater.cc:503] Master 127.0.98.62:34787 was elected leader, sending a full tablet report...
I20260619 18:40:53.514403 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:53.507179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37365 (local address 127.0.0.1:43262)
0619 18:40:53.508745 (+ 1566us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:53.508794 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:53.508933 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:53.510438 (+ 1505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:53.510445 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:53.510472 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:53.510617 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:53.510624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.511980 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.511983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:53.512834 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:53.512843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.513857 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.513860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:53.514052 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:53.514087 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 18:40:53.514183 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 18:40:53.514287 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1392,"thread_start_us":108,"threads_started":1}
I20260619 18:40:53.514441 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:53.508022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.0.98.1:37365)
0619 18:40:53.508471 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:53.508482 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:53.509990 (+ 1508us) server_negotiation.cc:407] Connection header received
0619 18:40:53.510067 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:53.510073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:53.510261 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:53.510424 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:53.510754 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.511850 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.512952 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.513711 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.513821 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:53.513986 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:53.514221 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 18:40:53.514315 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":100,"threads_started":1}
I20260619 18:40:53.515457 16464 tablet_service.cc:1514] Processing DeleteTablet for tablet 54c94f945e314548b5c2f51eb2366848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43262
I20260619 18:40:53.516341 16539 tablet_replica.cc:330] stopping tablet replica
I20260619 18:40:53.516635 16539 raft_consensus.cc:2229] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:53.516867 16539 raft_consensus.cc:2258] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:53.517805 16539 ts_tablet_manager.cc:1905] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:53.521270 16539 ts_tablet_manager.cc:1918] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:40:53.521348 16539 log.cc:1196] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal/wals/54c94f945e314548b5c2f51eb2366848
I20260619 18:40:53.521576 16539 ts_tablet_manager.cc:1939] T 54c94f945e314548b5c2f51eb2366848 P 77e1fea46ca94640bca652336c0a1f84: Deleting consensus metadata
I20260619 18:40:53.523981 16326 catalog_manager.cc:4841] TS 77e1fea46ca94640bca652336c0a1f84 (127.0.98.1:37365): tablet 54c94f945e314548b5c2f51eb2366848 (table test-table [id=4f559636b3c3407e8ebe4747fca23f97]) successfully deleted
W20260619 18:40:53.614151 16534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:53.614423 16534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:53.614483 16534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:53.618544 16534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:53.618604 16534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:53.618685 16534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:40:53.624058 16534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35335
--tserver_master_addrs=127.0.98.62:34787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:53.625116 16534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:53.626194 16534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:53.635689 16544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:53.635746 16534 server_base.cc:1034] running on GCE node
W20260619 18:40:53.635663 16545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:53.635679 16547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:53.636286 16534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:53.636909 16534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:53.638063 16534 hybrid_clock.cc:648] HybridClock initialized: now 1781894453638046 us; error 27 us; skew 500 ppm
Jun 19 18:40:53 dist-test-slave-rxv6 krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:53.640946 16534 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:40:53.641858 16534 webserver.cc:460] Webserver started at http://127.0.98.2:35335/ using document root <none> and password file <none>
I20260619 18:40:53.642513 16534 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:53.642592 16534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:53.646661 16534 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:40:53.648911 16554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:53.650285 16534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:40:53.650405 16534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "6632af5cdb994949abff101dc7f9c38c"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "30e26f921216a26be57ca43b0accba38"
server_key_iv: "fa7247a32ca3350617b8f24145354e7d"
server_key_version: "encryptionkey@0"
I20260619 18:40:53.650954 16534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:53.675109 16534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:53.678406 16534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:53.678576 16534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:53.679178 16534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:53.680240 16561 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 18:40:53.684217 16534 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 18:40:53.684283 16534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260619 18:40:53.684336 16534 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 18:40:53.686029 16534 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 18:40:53.686076 16534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260619 18:40:53.686342 16561 tablet_bootstrap.cc:492] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: Bootstrap starting.
I20260619 18:40:53.692030 16561 log.cc:824] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:53.695142 16534 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34287
I20260619 18:40:53.695171 16669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34287 every 8 connection(s)
I20260619 18:40:53.695953 16534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:40:53.700158 16561 tablet_bootstrap.cc:492] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:40:53.700488 16561 tablet_bootstrap.cc:492] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: Bootstrap complete.
I20260619 18:40:53.700776 16561 ts_tablet_manager.cc:1397] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260619 18:40:53.703104 16561 raft_consensus.cc:348] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } }
I20260619 18:40:53.703517 16561 raft_consensus.cc:729] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6632af5cdb994949abff101dc7f9c38c, State: Initialized, Role: FOLLOWER
I20260619 18:40:53.703918 16561 consensus_queue.cc:260] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } }
I20260619 18:40:53.704753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 16534
I20260619 18:40:53.704862 16561 ts_tablet_manager.cc:1428] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
Jun 19 18:40:53 dist-test-slave-rxv6 krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:53.705195 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39585
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34787
--webserver_port=39059
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:40:53.709033 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:53.697754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45575 (local address 127.0.98.62:34787)
0619 18:40:53.697870 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:53.697874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:53.698527 (+ 653us) server_negotiation.cc:407] Connection header received
0619 18:40:53.699534 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:53.699538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:53.699600 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:53.699679 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:53.700934 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.701464 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.702647 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.702874 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.702961 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:53.705743 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:53.705746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:53.705748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:53.705775 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:53.707114 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:53.707588 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:53.707591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:53.707592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:53.707668 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:53.707974 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:53.707977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:53.707979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:53.708159 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:53.708246 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:53.708729 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 18:40:53.708866 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:40:53.709805 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:53.697878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:45575)
0619 18:40:53.698381 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:53.698421 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:53.699336 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:53.699824 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:53.699831 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:53.700223 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:53.700781 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:53.700795 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.701609 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.701612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:53.702531 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:53.702541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.702970 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.702974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:53.703215 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:53.704049 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:53.704069 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:53.705583 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:53.707227 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:53.707232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:53.707246 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:53.707487 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:53.707750 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:53.707753 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:53.707755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:53.707888 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:53.708260 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:53.708266 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:53.708506 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 18:40:53.708673 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 18:40:53.708983 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":141,"threads_started":1}
I20260619 18:40:53.711089 16670 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34787
I20260619 18:40:53.711303 16670 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:53.711792 16670 heartbeater.cc:511] Master 127.0.98.62:34787 requested a full tablet report, sending...
I20260619 18:40:53.713378 16342 ts_manager.cc:194] Registered new tserver with Master: 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287)
I20260619 18:40:53.714540 16342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45575
I20260619 18:40:53.716590 16670 heartbeater.cc:503] Master 127.0.98.62:34787 was elected leader, sending a full tablet report...
Jun 19 18:40:53 dist-test-slave-rxv6 krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 18:40:53.723974 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:53.714282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34287 (local address 127.0.0.1:56982)
0619 18:40:53.714450 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:53.714463 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:53.714529 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:53.715232 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:53.715235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:53.715682 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:53.715782 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:53.715789 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.717227 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.717230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:53.718004 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:53.718013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.718738 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.718751 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:53.718812 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:53.719344 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:53.719355 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:53.720621 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:53.722596 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:53.722605 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:53.722609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:53.722817 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:53.723072 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:53.723074 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:53.723075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:53.723142 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:53.723570 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:53.723574 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:53.723666 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 18:40:53.723753 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 18:40:53.723859 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 18:40:53.724172 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:53.714388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56982 (local address 127.0.98.2:34287)
0619 18:40:53.714805 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:53.714815 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:53.714836 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:40:53.714898 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:53.714904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:53.715094 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:53.715220 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:53.715894 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.717110 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.718100 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.718637 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.718738 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:53.720748 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:53.720757 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:53.720770 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:53.720794 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:53.722476 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:53.722910 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:53.722915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:53.722918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:53.722999 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:53.723209 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:53.723212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:53.723214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:53.723478 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:53.723597 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:53.723839 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 18:40:53.723995 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":332,"spinlock_wait_cycles":4352,"thread_start_us":97,"threads_started":1}
I20260619 18:40:53.725148 16603 tablet_service.cc:1514] Processing DeleteTablet for tablet 54c94f945e314548b5c2f51eb2366848 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56982
I20260619 18:40:53.726193 16676 tablet_replica.cc:330] stopping tablet replica
I20260619 18:40:53.726464 16676 raft_consensus.cc:2229] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:53.726709 16676 raft_consensus.cc:2258] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:53.727574 16676 ts_tablet_manager.cc:1905] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:53.731101 16676 ts_tablet_manager.cc:1918] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:40:53.731185 16676 log.cc:1196] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/54c94f945e314548b5c2f51eb2366848
I20260619 18:40:53.731427 16676 ts_tablet_manager.cc:1939] T 54c94f945e314548b5c2f51eb2366848 P 6632af5cdb994949abff101dc7f9c38c: Deleting consensus metadata
I20260619 18:40:53.732889 16328 catalog_manager.cc:4841] TS 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287): tablet 54c94f945e314548b5c2f51eb2366848 (table test-table [id=4f559636b3c3407e8ebe4747fca23f97]) successfully deleted
W20260619 18:40:53.819525 16674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:40:53.819795 16674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:40:53.819839 16674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:40:53.823917 16674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:40:53.823987 16674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:40:53.824046 16674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:40:53.829403 16674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39059
--tserver_master_addrs=127.0.98.62:34787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:40:53.830533 16674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:40:53.831550 16674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:40:53.841369 16682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:40:53.841337 16681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:53.841392 16674 server_base.cc:1034] running on GCE node
W20260619 18:40:53.841514 16684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:40:53.842039 16674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:40:53.842687 16674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:40:53.843895 16674 hybrid_clock.cc:648] HybridClock initialized: now 1781894453843877 us; error 32 us; skew 500 ppm
Jun 19 18:40:53 dist-test-slave-rxv6 krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:40:53.846750 16674 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:40:53.847625 16674 webserver.cc:460] Webserver started at http://127.0.98.3:39059/ using document root <none> and password file <none>
I20260619 18:40:53.848165 16674 fs_manager.cc:359] Metadata directory not provided
I20260619 18:40:53.848248 16674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:40:53.852212 16674 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 18:40:53.854420 16691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:53.855819 16674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:40:53.855926 16674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "aad822701eb844d18769f67d29d5a67a"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "682ce07c64c353cbb4e1ee87a6f6e887"
server_key_iv: "0e223dff2e5c894490922c218d5ddac9"
server_key_version: "encryptionkey@0"
I20260619 18:40:53.856452 16674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:40:53.872409 16674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:40:53.875442 16674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:40:53.875617 16674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:40:53.876209 16674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:40:53.876995 16674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:40:53.877043 16674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:53.877094 16674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:40:53.877122 16674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:40:53.885403 16674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39585
I20260619 18:40:53.885428 16804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39585 every 8 connection(s)
I20260619 18:40:53.886373 16674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:40:53.891156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 16674
Jun 19 18:40:53 dist-test-slave-rxv6 krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:40:53.899626 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:53.891816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59900 (local address 127.0.98.62:34787)
0619 18:40:53.892101 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:53.892104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:53.892173 (+ 69us) server_negotiation.cc:407] Connection header received
0619 18:40:53.892287 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:53.892291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:53.892346 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:53.892430 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:53.893079 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.893677 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.894690 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.895015 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.895126 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:53.896499 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:53.896502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:53.896504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:53.896580 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:53.898055 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:53.898419 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:53.898423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:53.898425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:53.898511 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:53.898771 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:53.898774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:53.898776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:53.898976 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:53.899210 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:53.899317 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 18:40:53.899438 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":67,"threads_started":1}
I20260619 18:40:53.899989 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:53.888128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49523 (local address 127.0.98.62:34787)
0619 18:40:53.888267 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:53.888272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:53.889310 (+ 1038us) server_negotiation.cc:407] Connection header received
0619 18:40:53.890127 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:53.890130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:53.890177 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:53.890250 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:53.891385 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.892066 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.893380 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.893643 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.893720 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:53.896436 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:53.896440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:53.896443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:53.896469 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:53.897944 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:53.898475 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:53.898478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:53.898479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:53.898555 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:53.898894 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:53.898896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:53.898897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:53.899056 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:53.899167 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:53.899743 (+ 576us) server_negotiation.cc:299] Negotiation successful
0619 18:40:53.899847 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":48}
I20260619 18:40:53.901357 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:53.888495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:49523)
0619 18:40:53.889169 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:53.889207 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:53.889946 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:53.890392 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:53.890398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:53.890747 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:40:53.891225 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:53.891238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.892201 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.892204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:53.893263 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:53.893274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.893783 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.893788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:53.893963 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:53.894522 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:40:53.894537 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:40:53.896274 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:40:53.898082 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:53.898088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:53.898104 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:53.898380 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:53.898643 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:40:53.898646 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:40:53.898648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:40:53.898806 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:40:53.899155 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:40:53.899162 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:40:53.899490 (+ 328us) client_negotiation.cc:769] Sending connection context
0619 18:40:53.899688 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 18:40:53.900349 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":108,"threads_started":1}
I20260619 18:40:53.902415 16805 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34787
I20260619 18:40:53.902657 16805 heartbeater.cc:464] Registering TS with master...
I20260619 18:40:53.903119 16805 heartbeater.cc:511] Master 127.0.98.62:34787 requested a full tablet report, sending...
I20260619 18:40:53.904078 16342 ts_manager.cc:194] Registered new tserver with Master: aad822701eb844d18769f67d29d5a67a (127.0.98.3:39585)
I20260619 18:40:53.904620 16342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49523
I20260619 18:40:53.912995 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:40:53.917061 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:53.913612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59912 (local address 127.0.98.62:34787)
0619 18:40:53.913752 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:53.913755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:53.913896 (+ 141us) server_negotiation.cc:407] Connection header received
0619 18:40:53.914175 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:53.914177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:53.914277 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:53.914359 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:53.914598 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.915118 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.915969 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.916194 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.916272 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:53.916615 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:53.916751 (+ 136us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 18:40:53.916954 (+ 203us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59912: token expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 18:40:53.917135 16532 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59912: token expired
W20260619 18:40:53.917472 16270 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:34787)
I20260619 18:40:53.924679 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:53.917786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.0.98.62:34787)
0619 18:40:53.917887 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:53.917890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:53.917901 (+ 11us) server_negotiation.cc:407] Connection header received
0619 18:40:53.917999 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:53.918002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:53.918044 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:53.918097 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:40:53.918688 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.919176 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.919944 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.920170 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.920244 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:53.921485 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:40:53.921488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:40:53.921490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:40:53.921511 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:40:53.922915 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:53.923414 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:53.923418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:53.923420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:53.923512 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:40:53.923773 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:40:53.923777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:40:53.923778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:40:53.923986 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:40:53.924112 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:53.924339 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 18:40:53.924444 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 18:40:53.927245 16342 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:40:53.930716 16342 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:40:53.938495 16603 tablet_service.cc:1467] Processing CreateTablet for tablet a3d6f1eb21ae417cb44182776227f5ac (DEFAULT_TABLE table=test-table [id=6708176fe1974bfc80543588bb69cf88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:53.938495 16464 tablet_service.cc:1467] Processing CreateTablet for tablet a3d6f1eb21ae417cb44182776227f5ac (DEFAULT_TABLE table=test-table [id=6708176fe1974bfc80543588bb69cf88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:53.939138 16464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d6f1eb21ae417cb44182776227f5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:53.939139 16603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d6f1eb21ae417cb44182776227f5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:40:53.941881 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:53.937780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.0.98.3:39585)
0619 18:40:53.938232 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:53.938243 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:53.938264 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:40:53.938330 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:53.938335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:53.938504 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:53.938650 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:53.938898 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.939666 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.940661 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.941297 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.941396 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:53.941550 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:53.941646 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 18:40:53.941744 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":198,"threads_started":1}
I20260619 18:40:53.941870 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:53.937504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39585 (local address 127.0.0.1:45894)
0619 18:40:53.937686 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:53.937702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:53.937786 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:53.938642 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:53.938646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:53.938663 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:53.938787 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:53.938794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.939774 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.939777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:53.940532 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:53.940540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:53.941410 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:53.941413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:53.941509 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:53.941540 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:40:53.941614 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 18:40:53.941727 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 18:40:53.942005 16561 tablet_bootstrap.cc:492] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c: Bootstrap starting.
I20260619 18:40:53.942770 16422 tablet_bootstrap.cc:492] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84: Bootstrap starting.
I20260619 18:40:53.943033 16561 tablet_bootstrap.cc:654] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:53.943619 16422 tablet_bootstrap.cc:654] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:53.944597 16561 tablet_bootstrap.cc:492] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c: No bootstrap required, opened a new log
I20260619 18:40:53.944679 16561 ts_tablet_manager.cc:1397] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 18:40:53.944686 16422 tablet_bootstrap.cc:492] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84: No bootstrap required, opened a new log
I20260619 18:40:53.944777 16422 ts_tablet_manager.cc:1397] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 18:40:53.945086 16561 raft_consensus.cc:348] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } }
I20260619 18:40:53.945186 16561 raft_consensus.cc:374] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:53.945225 16561 raft_consensus.cc:729] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6632af5cdb994949abff101dc7f9c38c, State: Initialized, Role: FOLLOWER
I20260619 18:40:53.945163 16422 raft_consensus.cc:348] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } }
I20260619 18:40:53.945271 16422 raft_consensus.cc:374] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:53.945303 16422 raft_consensus.cc:729] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e1fea46ca94640bca652336c0a1f84, State: Initialized, Role: FOLLOWER
I20260619 18:40:53.945348 16561 consensus_queue.cc:260] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } }
I20260619 18:40:53.945429 16422 consensus_queue.cc:260] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } }
I20260619 18:40:53.945611 16561 ts_tablet_manager.cc:1428] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 18:40:53.945678 16422 ts_tablet_manager.cc:1428] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 18:40:53.945394 16739 tablet_service.cc:1467] Processing CreateTablet for tablet a3d6f1eb21ae417cb44182776227f5ac (DEFAULT_TABLE table=test-table [id=6708176fe1974bfc80543588bb69cf88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:40:53.946530 16739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3d6f1eb21ae417cb44182776227f5ac. 1 dirs total, 0 dirs full, 0 dirs failed
W20260619 18:40:53.946813 16671 tablet.cc:2369] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:40:53.953609 16817 tablet_bootstrap.cc:492] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a: Bootstrap starting.
I20260619 18:40:53.955801 16817 tablet_bootstrap.cc:654] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a: Neither blocks nor log segments found. Creating new log.
I20260619 18:40:53.956828 16817 log.cc:824] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:40:53.958853 16817 tablet_bootstrap.cc:492] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a: No bootstrap required, opened a new log
I20260619 18:40:53.959038 16817 ts_tablet_manager.cc:1397] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:40:53.962075 16817 raft_consensus.cc:348] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } }
I20260619 18:40:53.962316 16817 raft_consensus.cc:374] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:40:53.962376 16817 raft_consensus.cc:729] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad822701eb844d18769f67d29d5a67a, State: Initialized, Role: FOLLOWER
I20260619 18:40:53.962786 16817 consensus_queue.cc:260] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } }
I20260619 18:40:53.963397 16805 heartbeater.cc:503] Master 127.0.98.62:34787 was elected leader, sending a full tablet report...
I20260619 18:40:53.963635 16817 ts_tablet_manager.cc:1428] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 18:40:53.982031 16531 tablet.cc:2369] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:40:54.015892 16536 raft_consensus.cc:482] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:40:54.016129 16536 raft_consensus.cc:504] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } }
I20260619 18:40:54.017288 16536 leader_election.cc:288] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aad822701eb844d18769f67d29d5a67a (127.0.98.3:39585), 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287)
I20260619 18:40:54.020951 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:54.017658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39945 (local address 127.0.98.3:39585)
0619 18:40:54.017789 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:54.017802 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:54.018039 (+ 237us) server_negotiation.cc:407] Connection header received
0619 18:40:54.018170 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:54.018173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:54.018217 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:54.018278 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:54.018506 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:54.019007 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:54.020064 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:54.020462 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:54.020561 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:54.020625 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:54.020761 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 18:40:54.020830 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:40:54.020974 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:54.017503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39585 (local address 127.0.98.1:39945)
0619 18:40:54.017736 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:54.017751 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:54.018073 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:54.018295 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:54.018298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:54.018312 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:54.018410 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:54.018415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:54.019107 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:54.019110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:54.019948 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:54.019956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:54.020566 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:54.020570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:54.020649 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:54.020673 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:40:54.020737 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 18:40:54.020822 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":19,"thread_start_us":77,"threads_started":1}
I20260619 18:40:54.020998 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:54.017881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38923 (local address 127.0.98.2:34287)
0619 18:40:54.018020 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:54.018024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:54.018038 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:40:54.018168 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:54.018171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:54.018221 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:54.018311 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:40:54.018555 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:54.019066 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:54.020082 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:54.020500 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:54.020601 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:54.020758 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:54.020791 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 18:40:54.020864 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:40:54.021000 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:54.017555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34287 (local address 127.0.98.1:38923)
0619 18:40:54.017899 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:54.017931 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 18:40:54.018035 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:40:54.018333 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:40:54.018336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:40:54.018352 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:40:54.018453 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:54.018459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:54.019168 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:54.019171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:54.019966 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:40:54.019974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:40:54.020608 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:40:54.020611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:40:54.020687 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:54.020708 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:40:54.020765 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 18:40:54.020833 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":45,"thread_start_us":73,"threads_started":1}
I20260619 18:40:54.021613 16759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d6f1eb21ae417cb44182776227f5ac" candidate_uuid: "77e1fea46ca94640bca652336c0a1f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad822701eb844d18769f67d29d5a67a" is_pre_election: true
I20260619 18:40:54.021605 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d6f1eb21ae417cb44182776227f5ac" candidate_uuid: "77e1fea46ca94640bca652336c0a1f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6632af5cdb994949abff101dc7f9c38c" is_pre_election: true
I20260619 18:40:54.021899 16759 raft_consensus.cc:2454] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e1fea46ca94640bca652336c0a1f84 in term 0.
I20260619 18:40:54.021899 16623 raft_consensus.cc:2454] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e1fea46ca94640bca652336c0a1f84 in term 0.
I20260619 18:40:54.022380 16418 leader_election.cc:302] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6632af5cdb994949abff101dc7f9c38c, 77e1fea46ca94640bca652336c0a1f84; no voters:
I20260619 18:40:54.022682 16536 raft_consensus.cc:2781] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:40:54.022780 16536 raft_consensus.cc:482] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:40:54.022943 16536 raft_consensus.cc:3037] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:54.023962 16536 raft_consensus.cc:504] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } }
I20260619 18:40:54.024292 16536 leader_election.cc:288] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 1 election: Requested vote from peers aad822701eb844d18769f67d29d5a67a (127.0.98.3:39585), 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287)
I20260619 18:40:54.024652 16623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d6f1eb21ae417cb44182776227f5ac" candidate_uuid: "77e1fea46ca94640bca652336c0a1f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6632af5cdb994949abff101dc7f9c38c"
I20260619 18:40:54.024730 16759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3d6f1eb21ae417cb44182776227f5ac" candidate_uuid: "77e1fea46ca94640bca652336c0a1f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aad822701eb844d18769f67d29d5a67a"
I20260619 18:40:54.024797 16623 raft_consensus.cc:3037] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:54.024833 16759 raft_consensus.cc:3037] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:40:54.026024 16759 raft_consensus.cc:2454] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e1fea46ca94640bca652336c0a1f84 in term 1.
I20260619 18:40:54.026024 16623 raft_consensus.cc:2454] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e1fea46ca94640bca652336c0a1f84 in term 1.
I20260619 18:40:54.026297 16418 leader_election.cc:302] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6632af5cdb994949abff101dc7f9c38c, 77e1fea46ca94640bca652336c0a1f84; no voters:
I20260619 18:40:54.026466 16536 raft_consensus.cc:2781] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:40:54.026661 16536 raft_consensus.cc:686] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 LEADER]: Becoming Leader. State: Replica: 77e1fea46ca94640bca652336c0a1f84, State: Running, Role: LEADER
I20260619 18:40:54.026903 16536 consensus_queue.cc:237] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } }
I20260619 18:40:54.029889 16341 catalog_manager.cc:5495] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e1fea46ca94640bca652336c0a1f84 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "77e1fea46ca94640bca652336c0a1f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } health_report { overall_health: UNKNOWN } } }
I20260619 18:40:54.049968 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:54.046567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43274 (local address 127.0.98.1:37365)
0619 18:40:54.046826 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 18:40:54.046833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 18:40:54.046845 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:40:54.046917 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:40:54.046919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:40:54.046972 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:40:54.047074 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:40:54.047272 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:54.047785 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:54.048599 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:40:54.048826 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:40:54.048904 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:40:54.049132 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:40:54.049577 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:40:54.049660 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 18:40:54.049754 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 18:40:54.049803 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
I20260619 18:40:54.054054 16623 raft_consensus.cc:1261] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c [term 1 FOLLOWER]: Refusing update from remote peer 77e1fea46ca94640bca652336c0a1f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:54.054059 16759 raft_consensus.cc:1261] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a [term 1 FOLLOWER]: Refusing update from remote peer 77e1fea46ca94640bca652336c0a1f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:40:54.054704 16822 consensus_queue.cc:1035] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:54.054868 16536 consensus_queue.cc:1035] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:40:54.063691 16827 mvcc.cc:205] Tried to move back new op lower bound from 7298639683798257664 to 7298639683696570368. Current Snapshot: MvccSnapshot[applied={T|T < 7298639683798257664 or (T in {7298639683798257664})}]
I20260619 18:40:54.068866 16341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:54.069056 16341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:40:54.071488 16341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Sending DeleteTablet for 3 replicas of tablet a3d6f1eb21ae417cb44182776227f5ac
I20260619 18:40:54.072280 16603 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d6f1eb21ae417cb44182776227f5ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56982
I20260619 18:40:54.072376 16739 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d6f1eb21ae417cb44182776227f5ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45894
I20260619 18:40:54.072446 16464 tablet_service.cc:1514] Processing DeleteTablet for tablet a3d6f1eb21ae417cb44182776227f5ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:40:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43262
I20260619 18:40:54.072777 16836 tablet_replica.cc:330] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84: stopping tablet replica
I20260619 18:40:54.072801 16676 tablet_replica.cc:330] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c: stopping tablet replica
I20260619 18:40:54.072908 16836 raft_consensus.cc:2229] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:40:54.072921 16676 raft_consensus.cc:2229] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:54.073046 16676 raft_consensus.cc:2258] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:54.073185 16836 raft_consensus.cc:2258] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:54.073429 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 16380
I20260619 18:40:54.073804 16676 ts_tablet_manager.cc:1905] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:54.074358 16837 tablet_replica.cc:330] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a: stopping tablet replica
I20260619 18:40:54.074668 16837 raft_consensus.cc:2229] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:40:54.074972 16837 raft_consensus.cc:2258] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:40:54.076278 16676 ts_tablet_manager.cc:1918] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:40:54.076363 16676 log.cc:1196] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/a3d6f1eb21ae417cb44182776227f5ac
I20260619 18:40:54.076555 16837 ts_tablet_manager.cc:1905] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:40:54.076612 16676 ts_tablet_manager.cc:1939] T a3d6f1eb21ae417cb44182776227f5ac P 6632af5cdb994949abff101dc7f9c38c: Deleting consensus metadata
I20260619 18:40:54.077335 16328 catalog_manager.cc:4841] TS 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287): tablet a3d6f1eb21ae417cb44182776227f5ac (table test-table [id=6708176fe1974bfc80543588bb69cf88]) successfully deleted
I20260619 18:40:54.078768 16837 ts_tablet_manager.cc:1918] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:40:54.078872 16837 log.cc:1196] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/a3d6f1eb21ae417cb44182776227f5ac
I20260619 18:40:54.079171 16837 ts_tablet_manager.cc:1939] T a3d6f1eb21ae417cb44182776227f5ac P aad822701eb844d18769f67d29d5a67a: Deleting consensus metadata
I20260619 18:40:54.080240 16326 catalog_manager.cc:4841] TS aad822701eb844d18769f67d29d5a67a (127.0.98.3:39585): tablet a3d6f1eb21ae417cb44182776227f5ac (table test-table [id=6708176fe1974bfc80543588bb69cf88]) successfully deleted
W20260619 18:40:54.081056 16326 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37365 (error 108)
I20260619 18:40:54.081777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 16534
I20260619 18:40:54.082011 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:40:54.081737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37365 (local address 127.0.0.1:43276)
0619 18:40:54.081856 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 18:40:54.081937 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 18:40:54.082451 16326 catalog_manager.cc:4584] TS 77e1fea46ca94640bca652336c0a1f84 (127.0.98.1:37365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3d6f1eb21ae417cb44182776227f5ac: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37365: connect: Connection refused (error 111)
I20260619 18:40:54.086544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 16674
I20260619 18:40:54.091343 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 16309
I20260619 18:41:14.097014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34787
--webserver_interface=127.0.98.62
--webserver_port=40223
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:14.222738 16839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:14.223085 16839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:14.223130 16839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:14.227303 16839 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 18:41:14.227375 16839 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 18:41:14.227389 16839 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 18:41:14.227402 16839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:14.227421 16839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:14.227438 16839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:14.227466 16839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:14.227507 16839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:14.233278 16839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:14.234764 16839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:14.236039 16839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:41:14.246102 16839 server_base.cc:1034] running on GCE node
W20260619 18:41:14.246048 16845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:14.246048 16847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:14.246053 16844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:14.246979 16839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:14.248294 16839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:14.249467 16839 hybrid_clock.cc:648] HybridClock initialized: now 1781894474249447 us; error 40 us; skew 500 ppm
Jun 19 18:41:14 dist-test-slave-rxv6 krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:14.252465 16839 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:41:14.253477 16839 webserver.cc:460] Webserver started at http://127.0.98.62:40223/ using document root <none> and password file <none>
I20260619 18:41:14.254148 16839 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:14.254218 16839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:14.258580 16839 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:41:14.260932 16854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:14.262423 16839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:41:14.262547 16839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "a0331d23065d4f039a422c25c02604bd"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "28c1db75494a831209b862452210fd88"
server_key_iv: "40c01d7b2938c50211f85709ff76ca90"
server_key_version: "encryptionkey@0"
I20260619 18:41:14.263156 16839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:14.273108 16839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:14.276507 16839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:14.276691 16839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:14.284560 16839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34787
I20260619 18:41:14.284868 16906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34787 every 8 connection(s)
I20260619 18:41:14.285804 16839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:14.291046 16907 sys_catalog.cc:263] Verifying existing consensus state
I20260619 18:41:14.293078 16907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Bootstrap starting.
I20260619 18:41:14.294697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 16839
I20260619 18:41:14.295249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37365
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34787
--webserver_port=44863
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:41:14.313764 16907 log.cc:824] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:14.326128 16907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:41:14.326570 16907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Bootstrap complete.
I20260619 18:41:14.331411 16907 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } }
I20260619 18:41:14.331878 16907 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0331d23065d4f039a422c25c02604bd, State: Initialized, Role: FOLLOWER
I20260619 18:41:14.332468 16907 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } }
I20260619 18:41:14.332613 16907 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:14.332669 16907 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:14.332772 16907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 2 FOLLOWER]: Advancing to term 3
I20260619 18:41:14.335005 16907 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } }
I20260619 18:41:14.335368 16907 leader_election.cc:302] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0331d23065d4f039a422c25c02604bd; no voters:
I20260619 18:41:14.335716 16907 leader_election.cc:288] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 18:41:14.335858 16911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 3 FOLLOWER]: Leader election won for term 3
I20260619 18:41:14.336114 16911 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [term 3 LEADER]: Becoming Leader. State: Replica: a0331d23065d4f039a422c25c02604bd, State: Running, Role: LEADER
I20260619 18:41:14.336467 16911 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } }
I20260619 18:41:14.337071 16907 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:14.338052 16913 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0331d23065d4f039a422c25c02604bd. Latest consensus state: current_term: 3 leader_uuid: "a0331d23065d4f039a422c25c02604bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } } }
I20260619 18:41:14.338183 16913 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:14.338486 16912 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a0331d23065d4f039a422c25c02604bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0331d23065d4f039a422c25c02604bd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34787 } } }
I20260619 18:41:14.338570 16912 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:14.339138 16920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:14.343823 16920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:14.344372 16920 catalog_manager.cc:1259] Loaded cluster ID: 824cfb8382de4a79a7f5fe465e7dc89f
I20260619 18:41:14.344420 16920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:14.346360 16920 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:14.346951 16920 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Loaded TSK: 0
I20260619 18:41:14.347015 16920 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a0331d23065d4f039a422c25c02604bd: Loaded TSK: 1
F20260619 18:41:14.347658 16920 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fee193aa9ad google::LogMessage::Fail() at ??:0
@ 0x7fee193ad88a google::LogMessage::SendToLog() at ??:0
@ 0x7fee193aa4de google::LogMessage::Flush() at ??:0
@ 0x7fee193aba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fee1c06e18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fee1c066f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fee23d0f7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fee23ce999c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fee23cea178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fee23d2bdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fee23d4a82b std::function<>::operator()() at ??:0
@ 0x7fee23ce9b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fee23ceaf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fee23ce7579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fee23d2b640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fee1b5fc3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fee1b5fcc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fee1b5fe3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fee1b5ed512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fee1cc316db start_thread at ??:0
@ 0x7fee17ea861f clone at ??:0
W20260619 18:41:14.456864 16909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:14.457278 16909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:14.457353 16909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:14.463505 16909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:14.463599 16909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:14.463698 16909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:41:14.471896 16909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44863
--tserver_master_addrs=127.0.98.62:34787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:14.473587 16909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:14.475143 16909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:14.488986 16954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:14.489131 16955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:14.488997 16957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:14.489934 16909 server_base.cc:1034] running on GCE node
I20260619 18:41:14.490424 16909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:14.491204 16909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:14.492395 16909 hybrid_clock.cc:648] HybridClock initialized: now 1781894474492370 us; error 44 us; skew 500 ppm
Jun 19 18:41:14 dist-test-slave-rxv6 krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:14.495468 16909 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:41:14.496515 16909 webserver.cc:460] Webserver started at http://127.0.98.1:44863/ using document root <none> and password file <none>
I20260619 18:41:14.497169 16909 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:14.497233 16909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:14.502278 16909 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:41:14.505199 16964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:14.506969 16909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:41:14.507118 16909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "77e1fea46ca94640bca652336c0a1f84"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "d06453d28f4a93803d70cd08448f8de7"
server_key_iv: "3b5fc911a7571303f8382efe67974f64"
server_key_version: "encryptionkey@0"
I20260619 18:41:14.507840 16909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:14.522945 16909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:14.526975 16909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:14.527210 16909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:14.527993 16909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:14.529381 16971 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 18:41:14.536398 16909 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 18:41:14.536559 16909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260619 18:41:14.536649 16909 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 18:41:14.539017 16909 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 18:41:14.539086 16909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260619 18:41:14.539494 16971 tablet_bootstrap.cc:492] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84: Bootstrap starting.
I20260619 18:41:14.550535 16909 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37365
I20260619 18:41:14.550781 17078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37365 every 8 connection(s)
I20260619 18:41:14.551761 16909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:41:14.554031 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 16909
I20260619 18:41:14.554561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34287
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34787
--webserver_port=35335
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:41:14.565344 16971 log.cc:824] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:14.576797 16971 tablet_bootstrap.cc:492] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 18:41:14.577353 16971 tablet_bootstrap.cc:492] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84: Bootstrap complete.
I20260619 18:41:14.578902 16971 ts_tablet_manager.cc:1397] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84: Time spent bootstrapping tablet: real 0.040s user 0.019s sys 0.019s
I20260619 18:41:14.582862 16971 raft_consensus.cc:348] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } }
I20260619 18:41:14.583498 16971 raft_consensus.cc:729] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e1fea46ca94640bca652336c0a1f84, State: Initialized, Role: FOLLOWER
I20260619 18:41:14.584177 16971 consensus_queue.cc:260] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } }
I20260619 18:41:14.585520 16971 ts_tablet_manager.cc:1428] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
W20260619 18:41:14.615768 17081 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:41:14.554340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:42375)
0619 18:41:14.556096 (+ 1756us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:14.556170 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:14.557461 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:14.614647 (+ 57186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1458,"thread_start_us":150,"threads_started":1}
W20260619 18:41:14.616382 17079 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34787 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34787: Client connection negotiation failed: client connection to 127.0.98.62:34787: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 18:41:14.617223 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.616931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:33343)
0619 18:41:14.617072 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:14.617151 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 18:41:14.617856 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.617662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:40121)
0619 18:41:14.617757 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:14.617803 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 18:41:14.618043 17079 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 18:41:14.683797 17082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:14.684188 17082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:14.684264 17082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:14.688445 17082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:14.688516 17082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:14.688604 17082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:41:14.694190 17082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35335
--tserver_master_addrs=127.0.98.62:34787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:14.695465 17082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:14.696678 17082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:14.707757 17089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:14.707865 17092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:14.707818 17082 server_base.cc:1034] running on GCE node
W20260619 18:41:14.707731 17090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:14.708426 17082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:14.709156 17082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:14.710336 17082 hybrid_clock.cc:648] HybridClock initialized: now 1781894474710315 us; error 32 us; skew 500 ppm
Jun 19 18:41:14 dist-test-slave-rxv6 krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:14.713217 17082 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:41:14.714289 17082 webserver.cc:460] Webserver started at http://127.0.98.2:35335/ using document root <none> and password file <none>
I20260619 18:41:14.714892 17082 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:14.714951 17082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:14.719244 17082 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 18:41:14.721585 17099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:14.723035 17082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:41:14.723143 17082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "6632af5cdb994949abff101dc7f9c38c"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "30e26f921216a26be57ca43b0accba38"
server_key_iv: "fa7247a32ca3350617b8f24145354e7d"
server_key_version: "encryptionkey@0"
I20260619 18:41:14.723783 17082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:14.737136 17082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:14.740192 17082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:14.740376 17082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:14.741037 17082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:14.741987 17082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:14.742033 17082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:14.742090 17082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:14.742120 17082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:14.751809 17082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34287
I20260619 18:41:14.751844 17212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34287 every 8 connection(s)
I20260619 18:41:14.752995 17082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:41:14.757218 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.755502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:54017)
0619 18:41:14.756255 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:14.756379 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":124,"threads_started":1}
W20260619 18:41:14.757663 17213 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34787 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34787: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
I20260619 18:41:14.758272 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.758059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:35825)
0619 18:41:14.758166 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:14.758211 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 18:41:14.758800 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.758631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:50921)
0619 18:41:14.758713 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:14.758750 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 18:41:14.758939 17213 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 18:41:14.762758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 17082
I20260619 18:41:14.763187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39585
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34787
--webserver_port=39059
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:14.886770 17216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:14.887032 17216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:14.887073 17216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:14.891094 17216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:14.891147 17216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:14.891209 17216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:41:14.896454 17216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39059
--tserver_master_addrs=127.0.98.62:34787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:14.897629 17216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:14.898790 17216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:14.909533 17222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:14.909533 17221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:14.909641 17224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:14.909799 17216 server_base.cc:1034] running on GCE node
I20260619 18:41:14.910390 17216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:14.911144 17216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:14.912302 17216 hybrid_clock.cc:648] HybridClock initialized: now 1781894474912280 us; error 33 us; skew 500 ppm
Jun 19 18:41:14 dist-test-slave-rxv6 krb5kdc[15762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:14.915513 17216 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:41:14.916463 17216 webserver.cc:460] Webserver started at http://127.0.98.3:39059/ using document root <none> and password file <none>
I20260619 18:41:14.917085 17216 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:14.917142 17216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:14.921514 17216 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:41:14.923885 17231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:14.925261 17216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:41:14.925369 17216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "aad822701eb844d18769f67d29d5a67a"
format_stamp: "Formatted at 2026-06-19 18:40:32 on dist-test-slave-rxv6"
server_key: "682ce07c64c353cbb4e1ee87a6f6e887"
server_key_iv: "0e223dff2e5c894490922c218d5ddac9"
server_key_version: "encryptionkey@0"
I20260619 18:41:14.926055 17216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:14.940529 17216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:14.943981 17216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:14.944187 17216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:14.944913 17216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:14.946013 17216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:14.946062 17216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:14.946111 17216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:14.946138 17216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:14.955739 17216 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39585
I20260619 18:41:14.955780 17344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39585 every 8 connection(s)
I20260619 18:41:14.956923 17216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:41:14.960345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 17216
I20260619 18:41:14.962100 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.959426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:57179)
0619 18:41:14.960548 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:14.960771 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":129,"threads_started":1}
W20260619 18:41:14.962668 17345 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34787 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34787: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
I20260619 18:41:14.963328 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.963085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:40209)
0619 18:41:14.963213 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:14.963263 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 18:41:14.963855 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:14.963686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:59739)
0619 18:41:14.963766 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:14.963810 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 18:41:14.963984 17345 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 18:41:15.619158 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:15.618708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:47633)
0619 18:41:15.618992 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:15.619052 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
I20260619 18:41:15.759895 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:15.759475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:36927)
0619 18:41:15.759731 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:15.759787 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260619 18:41:15.910013 17351 raft_consensus.cc:482] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:15.910207 17351 raft_consensus.cc:504] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } }
I20260619 18:41:15.911561 17351 leader_election.cc:288] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aad822701eb844d18769f67d29d5a67a (127.0.98.3:39585), 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287)
Jun 19 18:41:15 dist-test-slave-rxv6 krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 19 18:41:15 dist-test-slave-rxv6 krb5kdc[15762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 18:41:15.927628 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:15.911761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34287 (local address 127.0.98.1:56441)
0619 18:41:15.912080 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:15.912094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:15.912188 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:15.914214 (+ 2026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:15.914222 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:15.914864 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:15.915670 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:15.915692 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.917599 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.917601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:15.919154 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:15.919164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.920106 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.920110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:15.920392 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:15.921094 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:15.921113 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:15.923257 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:15.925182 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:15.925184 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:15.925186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:15.925455 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:15.925759 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:15.925762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:15.925764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:15.925944 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:15.926572 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:15.926578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:15.926808 (+ 230us) client_negotiation.cc:769] Sending connection context
0619 18:41:15.927019 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 18:41:15.927364 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":359,"spinlock_wait_cycles":8192,"thread_start_us":70,"threads_started":1}
I20260619 18:41:15.927670 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:15.911711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39585 (local address 127.0.98.1:37065)
0619 18:41:15.911881 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:15.911894 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:15.911997 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:15.914214 (+ 2217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:15.914222 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:15.914856 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:15.915670 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:15.915692 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.917595 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.917598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:15.919153 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:15.919164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.920106 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.920109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:15.920397 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:15.921083 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:15.921107 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:15.922929 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:15.925099 (+ 2170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:15.925105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:15.925118 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:15.925449 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:15.925759 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:15.925762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:15.925764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:15.925962 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:15.926572 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:15.926578 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:15.926810 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 18:41:15.927003 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 18:41:15.927450 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":424,"spinlock_wait_cycles":88320}
I20260619 18:41:15.927688 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:15.911911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37065 (local address 127.0.98.3:39585)
0619 18:41:15.912224 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:15.912240 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:15.912288 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:41:15.913233 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:15.913242 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:15.913736 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:15.914107 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:15.915849 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.917440 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.919278 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.919978 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.920134 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:15.923207 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:15.923214 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:15.923231 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:15.923264 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:15.924984 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:15.925554 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:15.925559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:15.925563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:15.925671 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:15.926065 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:15.926067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:15.926068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:15.926463 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:15.926582 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:15.927147 (+ 565us) server_negotiation.cc:299] Negotiation successful
0619 18:41:15.927415 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":65,"threads_started":1}
I20260619 18:41:15.928292 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:15.911952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56441 (local address 127.0.98.2:34287)
0619 18:41:15.912348 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:15.912368 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:15.912398 (+ 30us) server_negotiation.cc:407] Connection header received
0619 18:41:15.913233 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:15.913242 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:15.913736 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:15.914107 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:15.915849 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.917433 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.919273 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:15.919978 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:15.920134 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:15.923426 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:15.923431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:15.923441 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:15.923502 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:15.925074 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:15.925554 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:15.925559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:15.925563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:15.925672 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:15.926032 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:15.926035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:15.926037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:15.926466 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:15.926581 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:15.927173 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 18:41:15.927462 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":130,"threads_started":1}
W20260619 18:41:15.930557 16967 leader_election.cc:341] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287): Not found: Tablet not found: a3d6f1eb21ae417cb44182776227f5ac
W20260619 18:41:15.930716 16965 leader_election.cc:341] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aad822701eb844d18769f67d29d5a67a (127.0.98.3:39585): Not found: Tablet not found: a3d6f1eb21ae417cb44182776227f5ac
I20260619 18:41:15.930787 16965 leader_election.cc:302] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e1fea46ca94640bca652336c0a1f84; no voters: 6632af5cdb994949abff101dc7f9c38c, aad822701eb844d18769f67d29d5a67a
I20260619 18:41:15.931054 17351 raft_consensus.cc:2726] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:41:15.965112 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:15.964619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:52731)
0619 18:41:15.964923 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:15.964992 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":96,"threads_started":1}
I20260619 18:41:16.620438 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:16.619953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:59707)
0619 18:41:16.620265 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:16.620340 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
I20260619 18:41:16.761116 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:16.760707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:46253)
0619 18:41:16.760959 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:16.761021 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20260619 18:41:16.966328 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:16.965849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:42261)
0619 18:41:16.966144 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:16.966203 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":101,"threads_started":1}
I20260619 18:41:17.621946 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:17.621386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:54557)
0619 18:41:17.621713 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:17.621787 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":99,"threads_started":1}
I20260619 18:41:17.762434 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:17.761893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:57599)
0619 18:41:17.762224 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:17.762299 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":82,"threads_started":1}
I20260619 18:41:17.967785 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:17.967278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:42233)
0619 18:41:17.967596 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:17.967662 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":115,"threads_started":1}
I20260619 18:41:17.968494 17362 raft_consensus.cc:482] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:17.968634 17362 raft_consensus.cc:504] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } }
I20260619 18:41:17.969113 17362 leader_election.cc:288] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aad822701eb844d18769f67d29d5a67a (127.0.98.3:39585), 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287)
W20260619 18:41:17.970077 16965 leader_election.cc:341] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aad822701eb844d18769f67d29d5a67a (127.0.98.3:39585): Not found: Tablet not found: a3d6f1eb21ae417cb44182776227f5ac
W20260619 18:41:17.970199 16967 leader_election.cc:341] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287): Not found: Tablet not found: a3d6f1eb21ae417cb44182776227f5ac
I20260619 18:41:17.970263 16967 leader_election.cc:302] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e1fea46ca94640bca652336c0a1f84; no voters: 6632af5cdb994949abff101dc7f9c38c, aad822701eb844d18769f67d29d5a67a
I20260619 18:41:17.970417 17362 raft_consensus.cc:2726] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:41:18.623325 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:18.622835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:60041)
0619 18:41:18.623137 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:18.623221 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":85,"threads_started":1}
I20260619 18:41:18.763751 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:18.763284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:34383)
0619 18:41:18.763580 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:18.763646 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":82,"threads_started":1}
I20260619 18:41:18.969072 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:18.968610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:45125)
0619 18:41:18.968909 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:18.968968 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":93,"threads_started":1}
I20260619 18:41:19.624876 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:19.624333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:33303)
0619 18:41:19.624667 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:19.624728 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":94,"threads_started":1}
I20260619 18:41:19.765065 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:19.764578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:36863)
0619 18:41:19.764897 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:19.764953 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":97,"threads_started":1}
I20260619 18:41:19.970541 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:19.970070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:50379)
0619 18:41:19.970357 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:19.970422 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
I20260619 18:41:20.227316 17369 raft_consensus.cc:482] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:20.227492 17369 raft_consensus.cc:504] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } }
I20260619 18:41:20.228032 17369 leader_election.cc:288] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aad822701eb844d18769f67d29d5a67a (127.0.98.3:39585), 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287)
W20260619 18:41:20.228891 16965 leader_election.cc:341] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aad822701eb844d18769f67d29d5a67a (127.0.98.3:39585): Not found: Tablet not found: a3d6f1eb21ae417cb44182776227f5ac
W20260619 18:41:20.229017 16967 leader_election.cc:341] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287): Not found: Tablet not found: a3d6f1eb21ae417cb44182776227f5ac
I20260619 18:41:20.229106 16967 leader_election.cc:302] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e1fea46ca94640bca652336c0a1f84; no voters: 6632af5cdb994949abff101dc7f9c38c, aad822701eb844d18769f67d29d5a67a
I20260619 18:41:20.229251 17369 raft_consensus.cc:2726] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:41:20.626155 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:20.625659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:57237)
0619 18:41:20.625966 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:20.626029 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":94,"threads_started":1}
I20260619 18:41:20.766319 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:20.765844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:59481)
0619 18:41:20.766145 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:20.766203 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":114,"threads_started":1}
I20260619 18:41:20.971895 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:20.971388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:54833)
0619 18:41:20.971656 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:20.971731 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20260619 18:41:21.627542 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:21.626975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:33745)
0619 18:41:21.627315 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:21.627384 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":93,"threads_started":1}
I20260619 18:41:21.767649 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:21.767066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:58909)
0619 18:41:21.767472 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:21.767532 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":108,"threads_started":1}
I20260619 18:41:21.973296 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:21.972785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:39089)
0619 18:41:21.973121 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:21.973178 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":91,"threads_started":1}
I20260619 18:41:22.497632 17376 raft_consensus.cc:482] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:22.497818 17376 raft_consensus.cc:504] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } }
I20260619 18:41:22.498347 17376 leader_election.cc:288] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aad822701eb844d18769f67d29d5a67a (127.0.98.3:39585), 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287)
W20260619 18:41:22.499135 16967 leader_election.cc:341] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287): Not found: Tablet not found: a3d6f1eb21ae417cb44182776227f5ac
W20260619 18:41:22.499255 16965 leader_election.cc:341] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aad822701eb844d18769f67d29d5a67a (127.0.98.3:39585): Not found: Tablet not found: a3d6f1eb21ae417cb44182776227f5ac
I20260619 18:41:22.499342 16965 leader_election.cc:302] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e1fea46ca94640bca652336c0a1f84; no voters: 6632af5cdb994949abff101dc7f9c38c, aad822701eb844d18769f67d29d5a67a
I20260619 18:41:22.499483 17376 raft_consensus.cc:2726] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:41:22.628818 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:22.628359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:55683)
0619 18:41:22.628621 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:22.628696 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":108,"threads_started":1}
I20260619 18:41:22.769045 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:22.768524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:50341)
0619 18:41:22.768848 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:22.768913 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":130,"threads_started":1}
I20260619 18:41:22.974723 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:22.974140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:57499)
0619 18:41:22.974497 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:22.974566 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":113,"threads_started":1}
I20260619 18:41:23.630332 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:23.629749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:46165)
0619 18:41:23.630110 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:23.630185 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":137,"threads_started":1}
I20260619 18:41:23.770480 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:23.770000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:47139)
0619 18:41:23.770286 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:23.770349 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":93,"threads_started":1}
I20260619 18:41:23.976020 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:23.975547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:47399)
0619 18:41:23.975821 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:23.975883 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":86,"threads_started":1}
I20260619 18:41:24.631814 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:24.631367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:34797)
0619 18:41:24.631641 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:24.631704 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260619 18:41:24.771754 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:24.771302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:54893)
0619 18:41:24.771583 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:24.771648 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":95,"threads_started":1}
I20260619 18:41:24.977438 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:24.977000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:38675)
0619 18:41:24.977275 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:24.977336 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
I20260619 18:41:25.633276 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:25.632729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:41213)
0619 18:41:25.633081 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:25.633155 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":93,"threads_started":1}
I20260619 18:41:25.773023 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:25.772572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:43891)
0619 18:41:25.772842 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:25.772904 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":88,"threads_started":1}
I20260619 18:41:25.978647 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:25.978156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:43609)
0619 18:41:25.978455 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:25.978514 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":122,"threads_started":1}
I20260619 18:41:26.634778 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.634240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:47861)
0619 18:41:26.634542 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:26.634628 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
I20260619 18:41:26.774590 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.774043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:49625)
0619 18:41:26.774381 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:26.774456 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
I20260619 18:41:26.980065 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:26.979538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:39057)
0619 18:41:26.979872 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:26.979940 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":115,"threads_started":1}
I20260619 18:41:27.636238 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.635712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:55445)
0619 18:41:27.636024 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:27.636094 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":97,"threads_started":1}
I20260619 18:41:27.776006 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.775464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:35103)
0619 18:41:27.775798 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:27.775862 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":90,"threads_started":1}
I20260619 18:41:27.981464 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:27.980948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:32933)
0619 18:41:27.981268 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:27.981333 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":95,"threads_started":1}
I20260619 18:41:28.394995 17395 raft_consensus.cc:482] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:28.395166 17395 raft_consensus.cc:504] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aad822701eb844d18769f67d29d5a67a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39585 } } peers { permanent_uuid: "77e1fea46ca94640bca652336c0a1f84" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37365 } } peers { permanent_uuid: "6632af5cdb994949abff101dc7f9c38c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34287 } }
I20260619 18:41:28.395716 17395 leader_election.cc:288] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aad822701eb844d18769f67d29d5a67a (127.0.98.3:39585), 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287)
W20260619 18:41:28.396587 16967 leader_election.cc:341] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6632af5cdb994949abff101dc7f9c38c (127.0.98.2:34287): Not found: Tablet not found: a3d6f1eb21ae417cb44182776227f5ac
W20260619 18:41:28.396737 16965 leader_election.cc:341] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aad822701eb844d18769f67d29d5a67a (127.0.98.3:39585): Not found: Tablet not found: a3d6f1eb21ae417cb44182776227f5ac
I20260619 18:41:28.396835 16965 leader_election.cc:302] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77e1fea46ca94640bca652336c0a1f84; no voters: 6632af5cdb994949abff101dc7f9c38c, aad822701eb844d18769f67d29d5a67a
I20260619 18:41:28.397010 17395 raft_consensus.cc:2726] T a3d6f1eb21ae417cb44182776227f5ac P 77e1fea46ca94640bca652336c0a1f84 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 18:41:28.637840 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.637263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:47713)
0619 18:41:28.637632 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:28.637706 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":136,"threads_started":1}
I20260619 18:41:28.777503 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.776935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:34753)
0619 18:41:28.777273 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:28.777338 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":124,"threads_started":1}
I20260619 18:41:28.982911 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:28.982381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.3:33651)
0619 18:41:28.982702 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:28.982766 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260619 18:41:29.639554 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.639011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.1:36251)
0619 18:41:29.639353 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:29.639423 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":109,"threads_started":1}
I20260619 18:41:29.779394 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:29.778802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34787 (local address 127.0.98.2:44177)
0619 18:41:29.779195 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:29.779269 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":136,"threads_started":1}
I20260619 18:41:29.971247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 16909
I20260619 18:41:29.977633 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 17082
I20260619 18:41:29.983566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 17216
2026-06-19T18:41:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59119 ms)
[----------] 30 tests from SecurityITest (240114 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:30 dist-test-slave-rxv6 krb5kdc[17403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:30 dist-test-slave-rxv6 krb5kdc[17403](info): set up 2 sockets
Jun 19 18:41:30 dist-test-slave-rxv6 krb5kdc[17403](info): commencing operation
krb5kdc: starting...
W20260619 18:41:30.945648 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:31 dist-test-slave-rxv6 krb5kdc[17403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:41:31.065506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37555
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:31.184011 17416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:31.184314 17416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:31.184372 17416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:31.188449 17416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:31.188530 17416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:31.188571 17416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:31.188592 17416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:31.188613 17416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:31.194129 17416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:31.195266 17416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:31.196309 17416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:31.204890 17421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:31.204903 17416 server_base.cc:1034] running on GCE node
W20260619 18:41:31.204903 17422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:31.204921 17424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:31.205691 17416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:31.206909 17416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:31.208091 17416 hybrid_clock.cc:648] HybridClock initialized: now 1781894491208071 us; error 36 us; skew 500 ppm
Jun 19 18:41:31 dist-test-slave-rxv6 krb5kdc[17403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:31.216101 17416 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:41:31.217080 17416 webserver.cc:460] Webserver started at http://127.0.98.62:34331/ using document root <none> and password file <none>
I20260619 18:41:31.217669 17416 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:31.217737 17416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:31.217998 17416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:31.220110 17416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "98f659ce66e549948f96b6f629edb357"
format_stamp: "Formatted at 2026-06-19 18:41:31 on dist-test-slave-rxv6"
server_key: "103b55ee97c9b487127bb231e59f1f82"
server_key_iv: "bb42bc6c4b4316ac7f1908779dc214e5"
server_key_version: "encryptionkey@0"
I20260619 18:41:31.220674 17416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "98f659ce66e549948f96b6f629edb357"
format_stamp: "Formatted at 2026-06-19 18:41:31 on dist-test-slave-rxv6"
server_key: "103b55ee97c9b487127bb231e59f1f82"
server_key_iv: "bb42bc6c4b4316ac7f1908779dc214e5"
server_key_version: "encryptionkey@0"
I20260619 18:41:31.224638 17416 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 18:41:31.227277 17431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:31.228617 17416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:41:31.228745 17416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "98f659ce66e549948f96b6f629edb357"
format_stamp: "Formatted at 2026-06-19 18:41:31 on dist-test-slave-rxv6"
server_key: "103b55ee97c9b487127bb231e59f1f82"
server_key_iv: "bb42bc6c4b4316ac7f1908779dc214e5"
server_key_version: "encryptionkey@0"
I20260619 18:41:31.228852 17416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:31.239744 17416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:31.242882 17416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:31.243072 17416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:31.249608 17416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37555
I20260619 18:41:31.249653 17483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37555 every 8 connection(s)
I20260619 18:41:31.250564 17416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:31.251613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 17416
I20260619 18:41:31.251722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:31.252027 392 external_mini_cluster.cc:1428] Setting key 3a117fc4bde39ead3851981bcfb535a8
I20260619 18:41:31.253739 17484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:31.260666 17484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357: Bootstrap starting.
Jun 19 18:41:31 dist-test-slave-rxv6 krb5kdc[17403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:41:31.263341 17484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:31.264263 17484 log.cc:824] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:31.266727 17484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357: No bootstrap required, opened a new log
I20260619 18:41:31.266952 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.253351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43142 (local address 127.0.98.62:37555)
0619 18:41:31.253829 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:31.253848 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:31.253898 (+ 50us) server_negotiation.cc:407] Connection header received
0619 18:41:31.254735 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:31.254758 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:31.255178 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:31.255552 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:31.256261 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.257746 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.258693 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.259382 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.259539 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.262037 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:31.262044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:31.262060 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:31.262093 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:31.263965 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:31.264423 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:31.264429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:31.264435 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:31.264568 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:31.264841 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:31.264843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:31.264844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:31.265301 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:31.265479 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:31.265719 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 18:41:31.266035 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":111,"threads_started":1}
I20260619 18:41:31.270646 17484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f659ce66e549948f96b6f629edb357" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37555 } }
I20260619 18:41:31.270879 17484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:31.270924 17484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f659ce66e549948f96b6f629edb357, State: Initialized, Role: FOLLOWER
I20260619 18:41:31.271301 17484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f659ce66e549948f96b6f629edb357" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37555 } }
I20260619 18:41:31.271448 17484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:31.271513 17484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:31.271608 17484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:31.272599 17484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f659ce66e549948f96b6f629edb357" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37555 } }
I20260619 18:41:31.272917 17484 leader_election.cc:302] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98f659ce66e549948f96b6f629edb357; no voters:
I20260619 18:41:31.273201 17484 leader_election.cc:288] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:31.273317 17489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:31.273524 17489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [term 1 LEADER]: Becoming Leader. State: Replica: 98f659ce66e549948f96b6f629edb357, State: Running, Role: LEADER
I20260619 18:41:31.273823 17489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f659ce66e549948f96b6f629edb357" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37555 } }
I20260619 18:41:31.274406 17484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:31.275641 17491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98f659ce66e549948f96b6f629edb357" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f659ce66e549948f96b6f629edb357" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37555 } } }
I20260619 18:41:31.275664 17490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98f659ce66e549948f96b6f629edb357. Latest consensus state: current_term: 1 leader_uuid: "98f659ce66e549948f96b6f629edb357" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f659ce66e549948f96b6f629edb357" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37555 } } }
I20260619 18:41:31.275786 17490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:31.275786 17491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:31.276343 17499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:31.279347 17499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:31.285344 17499 catalog_manager.cc:1347] Generated new cluster ID: c5db8a7ab33f46b08f6387f7e6ff7289
I20260619 18:41:31.285441 17499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:31.302649 17499 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:31.303644 17499 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:31.308382 17499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357: Generated new TSK 0
I20260619 18:41:31.309064 17499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:41:31.338091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37555
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:31.448372 17510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:31.448665 17510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:31.448710 17510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:31.452862 17510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:31.452936 17510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:31.452998 17510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:41:31.458694 17510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:31.459707 17510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:31.460685 17510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:31.470052 17516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:31.470085 17515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:31.470108 17510 server_base.cc:1034] running on GCE node
W20260619 18:41:31.470024 17518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:31.470602 17510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:31.471222 17510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:31.472388 17510 hybrid_clock.cc:648] HybridClock initialized: now 1781894491472371 us; error 32 us; skew 500 ppm
Jun 19 18:41:31 dist-test-slave-rxv6 krb5kdc[17403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:31.475222 17510 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:41:31.476220 17510 webserver.cc:460] Webserver started at http://127.0.98.1:42125/ using document root <none> and password file <none>
I20260619 18:41:31.476785 17510 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:31.476853 17510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:31.477041 17510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:31.479126 17510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "95080837a1fe409280bb02f96d223e57"
format_stamp: "Formatted at 2026-06-19 18:41:31 on dist-test-slave-rxv6"
server_key: "13a870f7ec619a98effda4052f5e9e23"
server_key_iv: "c8e9bfbd275aefdabdb16ff843414f6e"
server_key_version: "encryptionkey@0"
I20260619 18:41:31.479669 17510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "95080837a1fe409280bb02f96d223e57"
format_stamp: "Formatted at 2026-06-19 18:41:31 on dist-test-slave-rxv6"
server_key: "13a870f7ec619a98effda4052f5e9e23"
server_key_iv: "c8e9bfbd275aefdabdb16ff843414f6e"
server_key_version: "encryptionkey@0"
I20260619 18:41:31.483273 17510 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:41:31.485708 17525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:31.486984 17510 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:41:31.487105 17510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "95080837a1fe409280bb02f96d223e57"
format_stamp: "Formatted at 2026-06-19 18:41:31 on dist-test-slave-rxv6"
server_key: "13a870f7ec619a98effda4052f5e9e23"
server_key_iv: "c8e9bfbd275aefdabdb16ff843414f6e"
server_key_version: "encryptionkey@0"
I20260619 18:41:31.487205 17510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:31.504992 17510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:31.508041 17510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:31.508225 17510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:31.508827 17510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:31.509670 17510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:31.509716 17510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:31.509768 17510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:31.509795 17510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:31.518677 17510 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34611
I20260619 18:41:31.518704 17638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34611 every 8 connection(s)
I20260619 18:41:31.519577 17510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:41:31.524466 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 17510
I20260619 18:41:31.524575 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:41:31.524884 392 external_mini_cluster.cc:1428] Setting key 39825addc64bb0b2c5d78e2f0574b409
Jun 19 18:41:31 dist-test-slave-rxv6 krb5kdc[17403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:41:31.533120 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.521379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48411 (local address 127.0.98.62:37555)
0619 18:41:31.521567 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:31.521572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:31.522468 (+ 896us) server_negotiation.cc:407] Connection header received
0619 18:41:31.523326 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:31.523328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:31.523375 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:31.523450 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:31.524673 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.525385 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.526633 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.526859 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.526948 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.529278 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:31.529282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:31.529285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:31.529314 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:31.530944 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:31.531425 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:31.531437 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:31.531439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:31.531532 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:31.531852 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:31.531855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:31.531857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:31.532098 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:31.532200 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:31.532806 (+ 606us) server_negotiation.cc:299] Negotiation successful
0619 18:41:31.532941 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 18:41:31.533995 17641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.521811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37555 (local address 127.0.98.1:48411)
0619 18:41:31.522322 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.522365 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:31.523137 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:31.523659 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:31.523666 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:31.524019 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:31.524496 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.524510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.525534 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.525538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.526514 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.526525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.527004 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.527008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.527169 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.527759 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:31.527776 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:31.529116 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:31.531065 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:31.531070 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:31.531083 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:31.531327 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:31.531628 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:31.531630 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:31.531632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:31.531769 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:31.532201 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:31.532207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:31.532508 (+ 301us) client_negotiation.cc:769] Sending connection context
0619 18:41:31.532707 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 18:41:31.532993 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":125,"threads_started":1}
I20260619 18:41:31.535326 17639 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37555
I20260619 18:41:31.535555 17639 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:31.536077 17639 heartbeater.cc:511] Master 127.0.98.62:37555 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 18:41:31.537700 17445 ts_manager.cc:194] Registered new tserver with Master: 95080837a1fe409280bb02f96d223e57 (127.0.98.1:34611)
I20260619 18:41:31.538955 17445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48411
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:41:31.552778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37555
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:31.663322 17644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:31.663604 17644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:31.663647 17644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:31.667779 17644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:31.667847 17644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:31.667927 17644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:41:31.673393 17644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:31.674551 17644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:31.675614 17644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:31.685128 17652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:31.685210 17649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:31.685144 17650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:31.685468 17644 server_base.cc:1034] running on GCE node
I20260619 18:41:31.685954 17644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:31.686614 17644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:31.687777 17644 hybrid_clock.cc:648] HybridClock initialized: now 1781894491687755 us; error 38 us; skew 500 ppm
Jun 19 18:41:31 dist-test-slave-rxv6 krb5kdc[17403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:31.690693 17644 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:41:31.691726 17644 webserver.cc:460] Webserver started at http://127.0.98.2:37313/ using document root <none> and password file <none>
I20260619 18:41:31.692276 17644 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:31.692337 17644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:31.692548 17644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:31.694710 17644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "5f291f00fcb74137b7a1e09d97e9dc52"
format_stamp: "Formatted at 2026-06-19 18:41:31 on dist-test-slave-rxv6"
server_key: "5ef5904f97bea4d76548e4f51e543c19"
server_key_iv: "722df5dd432799e9078b486e434b61db"
server_key_version: "encryptionkey@0"
I20260619 18:41:31.695242 17644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5f291f00fcb74137b7a1e09d97e9dc52"
format_stamp: "Formatted at 2026-06-19 18:41:31 on dist-test-slave-rxv6"
server_key: "5ef5904f97bea4d76548e4f51e543c19"
server_key_iv: "722df5dd432799e9078b486e434b61db"
server_key_version: "encryptionkey@0"
I20260619 18:41:31.699118 17644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 18:41:31.701719 17659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:31.703125 17644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:41:31.703256 17644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "5f291f00fcb74137b7a1e09d97e9dc52"
format_stamp: "Formatted at 2026-06-19 18:41:31 on dist-test-slave-rxv6"
server_key: "5ef5904f97bea4d76548e4f51e543c19"
server_key_iv: "722df5dd432799e9078b486e434b61db"
server_key_version: "encryptionkey@0"
I20260619 18:41:31.703361 17644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:31.715358 17644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:31.718405 17644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:31.718590 17644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:31.719149 17644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:31.719954 17644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:31.720000 17644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:31.720052 17644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:31.720080 17644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:31.728431 17644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34637
I20260619 18:41:31.728443 17772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34637 every 8 connection(s)
I20260619 18:41:31.729323 17644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
Jun 19 18:41:31 dist-test-slave-rxv6 krb5kdc[17403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:41:31.739112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 17644
I20260619 18:41:31.739234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:41:31.739626 392 external_mini_cluster.cc:1428] Setting key 74dfba65bd948efd4f62cedf347e1633
I20260619 18:41:31.743391 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.731185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34707 (local address 127.0.98.62:37555)
0619 18:41:31.731336 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:31.731340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:31.732230 (+ 890us) server_negotiation.cc:407] Connection header received
0619 18:41:31.733225 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:31.733228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:31.733288 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:31.733362 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:31.734636 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.735171 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.736281 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.736510 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.736599 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.739284 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:31.739289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:31.739291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:31.739324 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:31.740860 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:31.741548 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:31.741552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:31.741553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:31.741664 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:31.742112 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:31.742114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:31.742115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:31.742307 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:31.742387 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:31.743079 (+ 692us) server_negotiation.cc:299] Negotiation successful
0619 18:41:31.743223 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:41:31.744184 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.731571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37555 (local address 127.0.98.2:34707)
0619 18:41:31.732081 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.732127 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:31.733020 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:31.733546 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:31.733554 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:31.734000 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:31.734497 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.734512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.735288 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.735291 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.736157 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.736166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.736649 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.736653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.736816 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.737377 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:31.737393 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:31.739079 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:31.741005 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:31.741010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:31.741025 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:31.741389 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:31.741763 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:31.741766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:31.741768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:31.742022 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:31.742400 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:31.742407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:31.742764 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 18:41:31.742983 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 18:41:31.743280 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":123,"threads_started":1}
I20260619 18:41:31.745393 17773 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37555
I20260619 18:41:31.745677 17773 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:31.746335 17773 heartbeater.cc:511] Master 127.0.98.62:37555 requested a full tablet report, sending...
I20260619 18:41:31.747586 17445 ts_manager.cc:194] Registered new tserver with Master: 5f291f00fcb74137b7a1e09d97e9dc52 (127.0.98.2:34637)
I20260619 18:41:31.748359 17445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34707
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:41:31.768841 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37555
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:31.882737 17778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:31.882982 17778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:31.883024 17778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:31.887077 17778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:31.887135 17778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:31.887192 17778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:41:31.892486 17778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:31.893478 17778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:31.894467 17778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:31.903810 17783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:31.903805 17784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:31.903815 17778 server_base.cc:1034] running on GCE node
W20260619 18:41:31.903878 17786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:31.904505 17778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:31.905270 17778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:31.908499 17778 hybrid_clock.cc:648] HybridClock initialized: now 1781894491908455 us; error 59 us; skew 500 ppm
Jun 19 18:41:31 dist-test-slave-rxv6 krb5kdc[17403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:31.911474 17778 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:41:31.912580 17778 webserver.cc:460] Webserver started at http://127.0.98.3:35999/ using document root <none> and password file <none>
I20260619 18:41:31.913151 17778 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:31.913219 17778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:31.913403 17778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:31.915524 17778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "585a92fb5cc34e949fc97023bff9e477"
format_stamp: "Formatted at 2026-06-19 18:41:31 on dist-test-slave-rxv6"
server_key: "ff8f371a6c09f9cfe2c42782c124d186"
server_key_iv: "0bfb084d521607e1587009ebb120e881"
server_key_version: "encryptionkey@0"
I20260619 18:41:31.916046 17778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "585a92fb5cc34e949fc97023bff9e477"
format_stamp: "Formatted at 2026-06-19 18:41:31 on dist-test-slave-rxv6"
server_key: "ff8f371a6c09f9cfe2c42782c124d186"
server_key_iv: "0bfb084d521607e1587009ebb120e881"
server_key_version: "encryptionkey@0"
I20260619 18:41:31.919684 17778 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:41:31.922269 17793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:31.923508 17778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:41:31.923622 17778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "585a92fb5cc34e949fc97023bff9e477"
format_stamp: "Formatted at 2026-06-19 18:41:31 on dist-test-slave-rxv6"
server_key: "ff8f371a6c09f9cfe2c42782c124d186"
server_key_iv: "0bfb084d521607e1587009ebb120e881"
server_key_version: "encryptionkey@0"
I20260619 18:41:31.923731 17778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:31.942932 17778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:31.945936 17778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:31.946116 17778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:31.946683 17778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:31.947476 17778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:31.947525 17778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:31.947578 17778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:31.947602 17778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:31.955914 17778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35679
I20260619 18:41:31.955952 17906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35679 every 8 connection(s)
I20260619 18:41:31.956791 17778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:41:31.965597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 17778
I20260619 18:41:31.965726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
Jun 19 18:41:31 dist-test-slave-rxv6 krb5kdc[17403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:41:31.966056 392 external_mini_cluster.cc:1428] Setting key d5a51d304623d3e5c8ee0da8eb0efbac
I20260619 18:41:31.970109 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.958595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52163 (local address 127.0.98.62:37555)
0619 18:41:31.958727 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:31.958730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:31.959879 (+ 1149us) server_negotiation.cc:407] Connection header received
0619 18:41:31.960781 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:31.960786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:31.960844 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:31.960920 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:31.962134 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.962671 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.963780 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.964006 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.964095 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.966635 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:31.966639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:31.966641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:31.966672 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:31.968228 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:31.968727 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:31.968729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:31.968731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:31.968802 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:31.969057 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:31.969059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:31.969060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:31.969247 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:31.969319 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:31.969823 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 18:41:31.969957 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:41:31.970886 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.958946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37555 (local address 127.0.98.3:52163)
0619 18:41:31.959715 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:31.959757 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:31.960573 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:31.961102 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:31.961109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:31.961480 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:31.961999 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.962015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.962789 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.962792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.963664 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:31.963674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.964143 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.964147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:31.964328 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.964894 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:31.964911 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:31.966505 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:31.968328 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:31.968333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:31.968346 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:31.968649 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:31.968865 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:31.968867 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:31.968868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:31.968993 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:31.969328 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:31.969333 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:31.969610 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 18:41:31.969762 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 18:41:31.970076 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":116,"threads_started":1}
I20260619 18:41:31.972008 17907 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37555
I20260619 18:41:31.972231 17907 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:31.972735 17907 heartbeater.cc:511] Master 127.0.98.62:37555 requested a full tablet report, sending...
I20260619 18:41:31.973747 17445 ts_manager.cc:194] Registered new tserver with Master: 585a92fb5cc34e949fc97023bff9e477 (127.0.98.3:35679)
I20260619 18:41:31.974324 17445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52163
I20260619 18:41:31.979983 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:41:31.989202 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:31.982063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43144 (local address 127.0.98.62:37555)
0619 18:41:31.982190 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:31.982194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:31.982372 (+ 178us) server_negotiation.cc:407] Connection header received
0619 18:41:31.982530 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:31.982532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:31.982579 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:31.982685 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:31.983352 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.983898 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.984769 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:31.984978 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:31.985046 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:31.986438 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:31.986441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:31.986443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:31.986465 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:31.987599 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:31.988042 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:31.988045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:31.988046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:31.988135 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:31.988433 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:31.988436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:31.988437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:31.988666 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:31.988773 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:31.988981 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 18:41:31.989079 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:41:31.992193 17445 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:41:31.995306 17445 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:41:32.009363 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.003596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35679 (local address 127.0.0.1:60794)
0619 18:41:32.004068 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:32.004124 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:32.004256 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:32.005333 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:32.005339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:32.005364 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:32.005499 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:32.005505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.006780 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.006783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:32.008030 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:32.008039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.008813 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.008817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:32.008923 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.008970 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 18:41:32.009081 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 18:41:32.009210 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":115,"threads_started":1}
I20260619 18:41:32.009368 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.003793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60794 (local address 127.0.98.3:35679)
0619 18:41:32.004182 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:32.004193 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:32.004219 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:41:32.004851 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:32.004858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:32.005095 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:32.005241 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:32.005636 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.006658 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.008145 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.008689 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.008797 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.008954 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:32.009112 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 18:41:32.009210 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":181,"threads_started":1}
I20260619 18:41:32.009819 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.004342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34611 (local address 127.0.0.1:43622)
0619 18:41:32.004666 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:32.004679 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:32.004761 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:32.005695 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:32.005697 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:32.005715 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:32.005889 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:32.005897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.007512 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.007515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:32.008379 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:32.008388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.009427 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.009431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:32.009532 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.009555 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:41:32.009603 (+ 48us) client_negotiation.cc:240] Negotiation successful
0619 18:41:32.009669 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":49,"spinlock_wait_cycles":3712,"thread_start_us":97,"threads_started":1}
I20260619 18:41:32.010576 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.004524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43622 (local address 127.0.98.1:34611)
0619 18:41:32.005299 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:32.005316 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:32.005338 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:41:32.005412 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:32.005416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:32.005568 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:32.005721 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:32.006044 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.007278 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.008492 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.009310 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.010121 (+ 811us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.010266 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:32.010317 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 18:41:32.010413 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":98,"threads_started":1}
I20260619 18:41:32.011667 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.004653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34637 (local address 127.0.0.1:54918)
0619 18:41:32.004938 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:32.004948 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:32.005015 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:32.005719 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:32.005722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:32.005742 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:32.005935 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:32.005948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.007451 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.007455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:32.009227 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:32.009238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.010300 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.010304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:32.010563 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.010597 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 18:41:32.011434 (+ 837us) client_negotiation.cc:240] Negotiation successful
0619 18:41:32.011529 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":48,"spinlock_wait_cycles":896,"thread_start_us":89,"threads_started":1}
I20260619 18:41:32.011747 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.005030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.0.98.2:34637)
0619 18:41:32.005298 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:32.005309 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:32.005329 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:41:32.005396 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:32.005401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:32.005582 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:32.005942 (+ 360us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:32.006149 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.007312 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.009366 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.010183 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.010280 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.010398 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:32.011492 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0619 18:41:32.011604 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":84,"threads_started":1}
I20260619 18:41:32.012852 17841 tablet_service.cc:1467] Processing CreateTablet for tablet cc65b887d4574a70b230d90aa858d1e8 (DEFAULT_TABLE table=test-table [id=fb2e76e419ab47bcb18317eae88e2182]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:32.013211 17573 tablet_service.cc:1467] Processing CreateTablet for tablet cc65b887d4574a70b230d90aa858d1e8 (DEFAULT_TABLE table=test-table [id=fb2e76e419ab47bcb18317eae88e2182]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:32.013828 17841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc65b887d4574a70b230d90aa858d1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:32.014297 17573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc65b887d4574a70b230d90aa858d1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:32.014653 17707 tablet_service.cc:1467] Processing CreateTablet for tablet cc65b887d4574a70b230d90aa858d1e8 (DEFAULT_TABLE table=test-table [id=fb2e76e419ab47bcb18317eae88e2182]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:32.015663 17707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc65b887d4574a70b230d90aa858d1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:32.020537 17924 tablet_bootstrap.cc:492] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477: Bootstrap starting.
I20260619 18:41:32.022511 17926 tablet_bootstrap.cc:492] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52: Bootstrap starting.
I20260619 18:41:32.022704 17924 tablet_bootstrap.cc:654] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:32.023417 17925 tablet_bootstrap.cc:492] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57: Bootstrap starting.
I20260619 18:41:32.023738 17924 log.cc:824] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:32.024766 17926 tablet_bootstrap.cc:654] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:32.025481 17925 tablet_bootstrap.cc:654] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:32.025570 17924 tablet_bootstrap.cc:492] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477: No bootstrap required, opened a new log
I20260619 18:41:32.025799 17924 ts_tablet_manager.cc:1397] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 18:41:32.025959 17926 log.cc:824] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:32.026669 17925 log.cc:824] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:32.027976 17926 tablet_bootstrap.cc:492] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52: No bootstrap required, opened a new log
I20260619 18:41:32.028260 17926 ts_tablet_manager.cc:1397] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:41:32.028589 17925 tablet_bootstrap.cc:492] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57: No bootstrap required, opened a new log
I20260619 18:41:32.028764 17925 ts_tablet_manager.cc:1397] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:41:32.029217 17924 raft_consensus.cc:348] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585a92fb5cc34e949fc97023bff9e477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35679 } } peers { permanent_uuid: "95080837a1fe409280bb02f96d223e57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } } peers { permanent_uuid: "5f291f00fcb74137b7a1e09d97e9dc52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260619 18:41:32.029441 17924 raft_consensus.cc:374] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:32.029516 17924 raft_consensus.cc:729] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 585a92fb5cc34e949fc97023bff9e477, State: Initialized, Role: FOLLOWER
I20260619 18:41:32.029949 17924 consensus_queue.cc:260] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585a92fb5cc34e949fc97023bff9e477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35679 } } peers { permanent_uuid: "95080837a1fe409280bb02f96d223e57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } } peers { permanent_uuid: "5f291f00fcb74137b7a1e09d97e9dc52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260619 18:41:32.030558 17907 heartbeater.cc:503] Master 127.0.98.62:37555 was elected leader, sending a full tablet report...
I20260619 18:41:32.030896 17924 ts_tablet_manager.cc:1428] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:41:32.032059 17926 raft_consensus.cc:348] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585a92fb5cc34e949fc97023bff9e477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35679 } } peers { permanent_uuid: "95080837a1fe409280bb02f96d223e57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } } peers { permanent_uuid: "5f291f00fcb74137b7a1e09d97e9dc52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260619 18:41:32.032361 17926 raft_consensus.cc:374] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:32.032433 17926 raft_consensus.cc:729] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f291f00fcb74137b7a1e09d97e9dc52, State: Initialized, Role: FOLLOWER
I20260619 18:41:32.032888 17926 consensus_queue.cc:260] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585a92fb5cc34e949fc97023bff9e477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35679 } } peers { permanent_uuid: "95080837a1fe409280bb02f96d223e57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } } peers { permanent_uuid: "5f291f00fcb74137b7a1e09d97e9dc52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260619 18:41:32.032884 17925 raft_consensus.cc:348] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585a92fb5cc34e949fc97023bff9e477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35679 } } peers { permanent_uuid: "95080837a1fe409280bb02f96d223e57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } } peers { permanent_uuid: "5f291f00fcb74137b7a1e09d97e9dc52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260619 18:41:32.033133 17925 raft_consensus.cc:374] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:32.033197 17925 raft_consensus.cc:729] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95080837a1fe409280bb02f96d223e57, State: Initialized, Role: FOLLOWER
I20260619 18:41:32.033558 17773 heartbeater.cc:503] Master 127.0.98.62:37555 was elected leader, sending a full tablet report...
I20260619 18:41:32.033725 17925 consensus_queue.cc:260] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585a92fb5cc34e949fc97023bff9e477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35679 } } peers { permanent_uuid: "95080837a1fe409280bb02f96d223e57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } } peers { permanent_uuid: "5f291f00fcb74137b7a1e09d97e9dc52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260619 18:41:32.033900 17926 ts_tablet_manager.cc:1428] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:41:32.034430 17639 heartbeater.cc:503] Master 127.0.98.62:37555 was elected leader, sending a full tablet report...
I20260619 18:41:32.034673 17925 ts_tablet_manager.cc:1428] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260619 18:41:32.207685 17908 tablet.cc:2369] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:41:32.230479 17774 tablet.cc:2369] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:41:32.236508 17932 raft_consensus.cc:482] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:32.236753 17932 raft_consensus.cc:504] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585a92fb5cc34e949fc97023bff9e477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35679 } } peers { permanent_uuid: "95080837a1fe409280bb02f96d223e57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } } peers { permanent_uuid: "5f291f00fcb74137b7a1e09d97e9dc52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260619 18:41:32.237895 17932 leader_election.cc:288] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 585a92fb5cc34e949fc97023bff9e477 (127.0.98.3:35679), 5f291f00fcb74137b7a1e09d97e9dc52 (127.0.98.2:34637)
I20260619 18:41:32.242895 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.238691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53771 (local address 127.0.98.2:34637)
0619 18:41:32.238836 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:32.238841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:32.238943 (+ 102us) server_negotiation.cc:407] Connection header received
0619 18:41:32.239067 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:32.239069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:32.239107 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:32.239175 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:32.239396 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.240131 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.241585 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.242248 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.242361 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.242426 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:32.242724 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 18:41:32.242793 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 18:41:32.242893 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.238300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40505 (local address 127.0.98.3:35679)
0619 18:41:32.238416 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:32.238420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:32.238496 (+ 76us) server_negotiation.cc:407] Connection header received
0619 18:41:32.238732 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:32.238736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:32.238791 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:32.238876 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:32.239187 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.239941 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.241552 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.242253 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.242361 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.242425 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:32.242683 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 18:41:32.242759 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:41:32.242938 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.238530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34637 (local address 127.0.98.1:53771)
0619 18:41:32.238851 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:32.238863 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:32.238981 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:32.239197 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:32.239199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:32.239218 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:32.239306 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:32.239310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.240251 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.240254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:32.241454 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:32.241461 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.242402 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.242406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:32.242621 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.242640 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 18:41:32.242703 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 18:41:32.242755 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":95,"spinlock_wait_cycles":2688,"thread_start_us":109,"threads_started":1}
I20260619 18:41:32.242941 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.238193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35679 (local address 127.0.98.1:40505)
0619 18:41:32.238419 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:32.238434 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:32.238602 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:32.238902 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:32.238905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:32.238925 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:32.239082 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:32.239089 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.240067 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.240070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:32.241393 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:32.241405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.242402 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.242406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:32.242547 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.242576 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 18:41:32.242668 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 18:41:32.242755 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":2,"spinlock_wait_cycles":2816,"thread_start_us":71,"threads_started":1}
I20260619 18:41:32.243631 17861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc65b887d4574a70b230d90aa858d1e8" candidate_uuid: "95080837a1fe409280bb02f96d223e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "585a92fb5cc34e949fc97023bff9e477" is_pre_election: true
I20260619 18:41:32.243633 17727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc65b887d4574a70b230d90aa858d1e8" candidate_uuid: "95080837a1fe409280bb02f96d223e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f291f00fcb74137b7a1e09d97e9dc52" is_pre_election: true
I20260619 18:41:32.243937 17727 raft_consensus.cc:2454] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95080837a1fe409280bb02f96d223e57 in term 0.
I20260619 18:41:32.243937 17861 raft_consensus.cc:2454] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95080837a1fe409280bb02f96d223e57 in term 0.
I20260619 18:41:32.244407 17527 leader_election.cc:302] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f291f00fcb74137b7a1e09d97e9dc52, 95080837a1fe409280bb02f96d223e57; no voters:
I20260619 18:41:32.244697 17932 raft_consensus.cc:2781] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:41:32.244786 17932 raft_consensus.cc:482] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:41:32.244844 17932 raft_consensus.cc:3037] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:32.245996 17932 raft_consensus.cc:504] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585a92fb5cc34e949fc97023bff9e477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35679 } } peers { permanent_uuid: "95080837a1fe409280bb02f96d223e57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } } peers { permanent_uuid: "5f291f00fcb74137b7a1e09d97e9dc52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260619 18:41:32.246389 17932 leader_election.cc:288] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [CANDIDATE]: Term 1 election: Requested vote from peers 585a92fb5cc34e949fc97023bff9e477 (127.0.98.3:35679), 5f291f00fcb74137b7a1e09d97e9dc52 (127.0.98.2:34637)
I20260619 18:41:32.246865 17861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc65b887d4574a70b230d90aa858d1e8" candidate_uuid: "95080837a1fe409280bb02f96d223e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "585a92fb5cc34e949fc97023bff9e477"
I20260619 18:41:32.246910 17727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc65b887d4574a70b230d90aa858d1e8" candidate_uuid: "95080837a1fe409280bb02f96d223e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f291f00fcb74137b7a1e09d97e9dc52"
I20260619 18:41:32.247030 17727 raft_consensus.cc:3037] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:32.247037 17861 raft_consensus.cc:3037] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:32.248317 17861 raft_consensus.cc:2454] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95080837a1fe409280bb02f96d223e57 in term 1.
I20260619 18:41:32.248317 17727 raft_consensus.cc:2454] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95080837a1fe409280bb02f96d223e57 in term 1.
I20260619 18:41:32.248659 17527 leader_election.cc:302] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585a92fb5cc34e949fc97023bff9e477, 95080837a1fe409280bb02f96d223e57; no voters:
I20260619 18:41:32.248829 17932 raft_consensus.cc:2781] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:32.249042 17932 raft_consensus.cc:686] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [term 1 LEADER]: Becoming Leader. State: Replica: 95080837a1fe409280bb02f96d223e57, State: Running, Role: LEADER
I20260619 18:41:32.249361 17932 consensus_queue.cc:237] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585a92fb5cc34e949fc97023bff9e477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35679 } } peers { permanent_uuid: "95080837a1fe409280bb02f96d223e57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } } peers { permanent_uuid: "5f291f00fcb74137b7a1e09d97e9dc52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } }
I20260619 18:41:32.252519 17445 catalog_manager.cc:5495] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95080837a1fe409280bb02f96d223e57 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "95080837a1fe409280bb02f96d223e57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "585a92fb5cc34e949fc97023bff9e477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95080837a1fe409280bb02f96d223e57" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f291f00fcb74137b7a1e09d97e9dc52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 } health_report { overall_health: UNKNOWN } } }
W20260619 18:41:32.270869 17640 tablet.cc:2369] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:41:32.301762 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.298259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.0.98.1:34611)
0619 18:41:32.298399 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:32.298404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:32.298418 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:41:32.298512 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:32.298515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:32.298576 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:32.298669 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:41:32.298920 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.299438 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.300285 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.300529 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.300618 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.300940 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:41:32.301371 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:41:32.301483 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:32.301561 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 18:41:32.301622 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:41:32.308230 17727 raft_consensus.cc:1261] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52 [term 1 FOLLOWER]: Refusing update from remote peer 95080837a1fe409280bb02f96d223e57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:32.308230 17861 raft_consensus.cc:1261] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477 [term 1 FOLLOWER]: Refusing update from remote peer 95080837a1fe409280bb02f96d223e57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:32.309069 17935 consensus_queue.cc:1035] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "585a92fb5cc34e949fc97023bff9e477" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:32.309270 17932 consensus_queue.cc:1035] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f291f00fcb74137b7a1e09d97e9dc52" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:32.317543 17940 mvcc.cc:205] Tried to move back new op lower bound from 7298639840485748736 to 7298639840256352256. Current Snapshot: MvccSnapshot[applied={T|T < 7298639840485748736}]
I20260619 18:41:32.318632 17939 mvcc.cc:205] Tried to move back new op lower bound from 7298639840485748736 to 7298639840256352256. Current Snapshot: MvccSnapshot[applied={T|T < 7298639840485748736}]
I20260619 18:41:32.325306 17448 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:32.325538 17448 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:32.328400 17448 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98f659ce66e549948f96b6f629edb357: Sending DeleteTablet for 3 replicas of tablet cc65b887d4574a70b230d90aa858d1e8
I20260619 18:41:32.329296 17841 tablet_service.cc:1514] Processing DeleteTablet for tablet cc65b887d4574a70b230d90aa858d1e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60794
I20260619 18:41:32.329414 17573 tablet_service.cc:1514] Processing DeleteTablet for tablet cc65b887d4574a70b230d90aa858d1e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43622
I20260619 18:41:32.329573 17707 tablet_service.cc:1514] Processing DeleteTablet for tablet cc65b887d4574a70b230d90aa858d1e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54918
I20260619 18:41:32.330070 17949 tablet_replica.cc:330] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52: stopping tablet replica
I20260619 18:41:32.330068 17948 tablet_replica.cc:330] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57: stopping tablet replica
I20260619 18:41:32.330446 17949 raft_consensus.cc:2229] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:32.330482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 17510
I20260619 18:41:32.330476 17948 raft_consensus.cc:2229] T cc65b887d4574a70b230d90aa858d1e8 P 95080837a1fe409280bb02f96d223e57 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:41:32.330830 17949 raft_consensus.cc:2258] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:32.332444 17947 tablet_replica.cc:330] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477: stopping tablet replica
I20260619 18:41:32.332752 17949 ts_tablet_manager.cc:1905] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:32.332767 17947 raft_consensus.cc:2229] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:32.333068 17947 raft_consensus.cc:2258] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:32.334398 17947 ts_tablet_manager.cc:1905] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:32.335672 17949 ts_tablet_manager.cc:1918] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:32.335772 17949 log.cc:1196] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/cc65b887d4574a70b230d90aa858d1e8
I20260619 18:41:32.336102 17949 ts_tablet_manager.cc:1939] T cc65b887d4574a70b230d90aa858d1e8 P 5f291f00fcb74137b7a1e09d97e9dc52: Deleting consensus metadata
I20260619 18:41:32.336526 17947 ts_tablet_manager.cc:1918] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:32.336609 17947 log.cc:1196] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/cc65b887d4574a70b230d90aa858d1e8
I20260619 18:41:32.336853 17947 ts_tablet_manager.cc:1939] T cc65b887d4574a70b230d90aa858d1e8 P 585a92fb5cc34e949fc97023bff9e477: Deleting consensus metadata
I20260619 18:41:32.337499 17433 catalog_manager.cc:4841] TS 5f291f00fcb74137b7a1e09d97e9dc52 (127.0.98.2:34637): tablet cc65b887d4574a70b230d90aa858d1e8 (table test-table [id=fb2e76e419ab47bcb18317eae88e2182]) successfully deleted
I20260619 18:41:32.337852 17433 catalog_manager.cc:4841] TS 585a92fb5cc34e949fc97023bff9e477 (127.0.98.3:35679): tablet cc65b887d4574a70b230d90aa858d1e8 (table test-table [id=fb2e76e419ab47bcb18317eae88e2182]) successfully deleted
W20260619 18:41:32.338675 17433 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34611 (error 108)
I20260619 18:41:32.338716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 17644
I20260619 18:41:32.340245 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.339873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34611 (local address 127.0.0.1:43640)
0619 18:41:32.340085 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:32.340157 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34611: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260619 18:41:32.340404 17433 catalog_manager.cc:4584] TS 95080837a1fe409280bb02f96d223e57 (127.0.98.1:34611): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc65b887d4574a70b230d90aa858d1e8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34611: connect: Connection refused (error 111)
I20260619 18:41:32.344010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 17778
I20260619 18:41:32.348707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 17416
2026-06-19T18:41:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2362 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T18:41:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:32Z Disabled control of system clock
I20260619 18:41:32.377337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35985
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:41:32.486155 17954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:32.486441 17954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:32.486476 17954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:32.490619 17954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:32.490705 17954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:32.490720 17954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:32.490736 17954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:32.490749 17954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:32.495777 17954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:32.496882 17954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:32.498283 17954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:32.506903 17959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:32.506891 17962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:32.506893 17960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:32.507023 17954 server_base.cc:1034] running on GCE node
I20260619 18:41:32.507786 17954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:32.508951 17954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:32.510115 17954 hybrid_clock.cc:648] HybridClock initialized: now 1781894492510100 us; error 39 us; skew 500 ppm
I20260619 18:41:32.512042 17954 webserver.cc:460] Webserver started at http://127.0.98.62:39335/ using document root <none> and password file <none>
I20260619 18:41:32.512610 17954 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:32.512676 17954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:32.512864 17954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:32.515082 17954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "2e19fe680e63466a85a33abac1617094"
format_stamp: "Formatted at 2026-06-19 18:41:32 on dist-test-slave-rxv6"
server_key: "a8f1939963b062843ed95ffbfc85f516"
server_key_iv: "c65e9b0d3c115bdceaec69dc77a6d688"
server_key_version: "encryptionkey@0"
I20260619 18:41:32.515654 17954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e19fe680e63466a85a33abac1617094"
format_stamp: "Formatted at 2026-06-19 18:41:32 on dist-test-slave-rxv6"
server_key: "a8f1939963b062843ed95ffbfc85f516"
server_key_iv: "c65e9b0d3c115bdceaec69dc77a6d688"
server_key_version: "encryptionkey@0"
I20260619 18:41:32.519562 17954 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 18:41:32.522049 17968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:32.523332 17954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:41:32.523447 17954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "2e19fe680e63466a85a33abac1617094"
format_stamp: "Formatted at 2026-06-19 18:41:32 on dist-test-slave-rxv6"
server_key: "a8f1939963b062843ed95ffbfc85f516"
server_key_iv: "c65e9b0d3c115bdceaec69dc77a6d688"
server_key_version: "encryptionkey@0"
I20260619 18:41:32.523548 17954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:32.534272 17954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:32.535007 17954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:32.535199 17954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:32.542088 17954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35985
I20260619 18:41:32.542156 18020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35985 every 8 connection(s)
I20260619 18:41:32.543064 17954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:32.546471 18021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:32.552467 18021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094: Bootstrap starting.
I20260619 18:41:32.553223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 17954
I20260619 18:41:32.553347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:32.553598 392 external_mini_cluster.cc:1428] Setting key 82dbb9b3499a48ae14f375d1d6afdf3c
I20260619 18:41:32.555397 18021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:32.556358 18021 log.cc:824] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:32.558722 18021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094: No bootstrap required, opened a new log
I20260619 18:41:32.562897 18021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e19fe680e63466a85a33abac1617094" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35985 } }
I20260619 18:41:32.563125 18021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:32.563160 18021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e19fe680e63466a85a33abac1617094, State: Initialized, Role: FOLLOWER
I20260619 18:41:32.563472 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.554803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.0.98.62:35985)
0619 18:41:32.555774 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:32.555800 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:32.555851 (+ 51us) server_negotiation.cc:407] Connection header received
0619 18:41:32.556720 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:32.556749 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:32.557183 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:32.557517 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:32.557871 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.558894 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.560026 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.560561 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.560698 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.561555 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:32.561560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:32.561573 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:32.562016 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:32.562077 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:32.562123 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:32.562201 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:32.562347 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 18:41:32.562618 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":150,"threads_started":1}
I20260619 18:41:32.563617 18021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e19fe680e63466a85a33abac1617094" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35985 } }
I20260619 18:41:32.563771 18021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:32.563838 18021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:32.563938 18021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:32.564982 18021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e19fe680e63466a85a33abac1617094" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35985 } }
I20260619 18:41:32.565340 18021 leader_election.cc:302] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e19fe680e63466a85a33abac1617094; no voters:
I20260619 18:41:32.565670 18021 leader_election.cc:288] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:32.565783 18026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:32.566025 18026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [term 1 LEADER]: Becoming Leader. State: Replica: 2e19fe680e63466a85a33abac1617094, State: Running, Role: LEADER
I20260619 18:41:32.566381 18026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e19fe680e63466a85a33abac1617094" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35985 } }
I20260619 18:41:32.566869 18021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:32.567498 18026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e19fe680e63466a85a33abac1617094. Latest consensus state: current_term: 1 leader_uuid: "2e19fe680e63466a85a33abac1617094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e19fe680e63466a85a33abac1617094" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35985 } } }
I20260619 18:41:32.567621 18026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:32.567878 18027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e19fe680e63466a85a33abac1617094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e19fe680e63466a85a33abac1617094" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35985 } } }
I20260619 18:41:32.567979 18027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:32.568238 18033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:32.571552 18033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:32.578061 18033 catalog_manager.cc:1347] Generated new cluster ID: 9ff72db657804f9fa6115213ccf14aa4
I20260619 18:41:32.578155 18033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:32.587126 18033 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:32.588078 18033 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:32.592938 18033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094: Generated new TSK 0
I20260619 18:41:32.593626 18033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:32.602218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35985
--builtin_ntp_servers=127.0.98.20:41377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:41:32.715885 18045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:32.716126 18045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:32.716162 18045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:32.720201 18045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:32.720260 18045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:32.720314 18045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:41:32.725023 18045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:32.726099 18045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:32.727444 18045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:32.736819 18053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:32.736819 18051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:32.736814 18050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:32.736943 18045 server_base.cc:1034] running on GCE node
I20260619 18:41:32.737424 18045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:32.738055 18045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:32.739223 18045 hybrid_clock.cc:648] HybridClock initialized: now 1781894492739207 us; error 34 us; skew 500 ppm
I20260619 18:41:32.741142 18045 webserver.cc:460] Webserver started at http://127.0.98.1:37701/ using document root <none> and password file <none>
I20260619 18:41:32.741681 18045 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:32.741739 18045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:32.741950 18045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:32.743948 18045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "071789f4ee9941cab13c1b2b74dc9f75"
format_stamp: "Formatted at 2026-06-19 18:41:32 on dist-test-slave-rxv6"
server_key: "800a1d7ecd6d3cdd04e42cbf7071b7fc"
server_key_iv: "557cfe41d295372f191a97059b0824af"
server_key_version: "encryptionkey@0"
I20260619 18:41:32.744472 18045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "071789f4ee9941cab13c1b2b74dc9f75"
format_stamp: "Formatted at 2026-06-19 18:41:32 on dist-test-slave-rxv6"
server_key: "800a1d7ecd6d3cdd04e42cbf7071b7fc"
server_key_iv: "557cfe41d295372f191a97059b0824af"
server_key_version: "encryptionkey@0"
I20260619 18:41:32.748194 18045 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:41:32.750648 18059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:32.751916 18045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:41:32.752033 18045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "071789f4ee9941cab13c1b2b74dc9f75"
format_stamp: "Formatted at 2026-06-19 18:41:32 on dist-test-slave-rxv6"
server_key: "800a1d7ecd6d3cdd04e42cbf7071b7fc"
server_key_iv: "557cfe41d295372f191a97059b0824af"
server_key_version: "encryptionkey@0"
I20260619 18:41:32.752131 18045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:32.776289 18045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:32.776991 18045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:32.777172 18045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:32.777767 18045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:32.778750 18045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:32.778798 18045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:32.778851 18045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:32.778880 18045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:32.787032 18045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41047
I20260619 18:41:32.787066 18172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41047 every 8 connection(s)
I20260619 18:41:32.787984 18045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:41:32.789203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 18045
I20260619 18:41:32.789322 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:41:32.789613 392 external_mini_cluster.cc:1428] Setting key aa203754e74716f72ece06955a5b9dd6
I20260619 18:41:32.791611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35985
--builtin_ntp_servers=127.0.98.20:41377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:41:32.797677 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.789994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48639 (local address 127.0.98.62:35985)
0619 18:41:32.790150 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:32.790155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:32.791431 (+ 1276us) server_negotiation.cc:407] Connection header received
0619 18:41:32.792919 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:32.792922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:32.792978 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:32.793049 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:32.794020 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.794580 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.795760 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.795991 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.796078 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.797200 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:32.797203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:32.797205 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:32.797217 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:32.797238 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:32.797246 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:32.797318 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:32.797460 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 18:41:32.797527 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:41:32.798659 18175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.790503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35985 (local address 127.0.98.1:48639)
0619 18:41:32.791257 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:32.791312 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:32.792705 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:32.793234 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:32.793241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:32.793337 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:32.793862 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:32.793875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.794709 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.794712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:32.795632 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:32.795643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.796133 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.796138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:32.796295 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.796885 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:41:32.796886 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:32.796939 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:41:32.797011 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:32.797320 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:32.797331 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:32.797336 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 18:41:32.797447 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 18:41:32.797748 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":119,"threads_started":1}
I20260619 18:41:32.799979 18173 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35985
I20260619 18:41:32.800259 18173 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:32.800884 18173 heartbeater.cc:511] Master 127.0.98.62:35985 requested a full tablet report, sending...
I20260619 18:41:32.802585 17985 ts_manager.cc:194] Registered new tserver with Master: 071789f4ee9941cab13c1b2b74dc9f75 (127.0.98.1:41047)
I20260619 18:41:32.804150 17985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:48639
W20260619 18:41:32.903493 18176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:32.903810 18176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:32.903847 18176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:32.907866 18176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:32.907938 18176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:32.908015 18176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:41:32.913667 18176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:32.914762 18176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:32.916142 18176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:32.925735 18184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:32.925772 18182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:32.925823 18176 server_base.cc:1034] running on GCE node
W20260619 18:41:32.925804 18181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:32.926414 18176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:32.927141 18176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:32.928296 18176 hybrid_clock.cc:648] HybridClock initialized: now 1781894492928272 us; error 31 us; skew 500 ppm
I20260619 18:41:32.930347 18176 webserver.cc:460] Webserver started at http://127.0.98.2:44765/ using document root <none> and password file <none>
I20260619 18:41:32.930943 18176 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:32.931006 18176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:32.931181 18176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:32.933439 18176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "45ba2c5f1f4b453bbe65c4760c3edac5"
format_stamp: "Formatted at 2026-06-19 18:41:32 on dist-test-slave-rxv6"
server_key: "7eadc47e43e7315757508455f23557ba"
server_key_iv: "53a843b36ec2e60d0ea26f5c38eb3507"
server_key_version: "encryptionkey@0"
I20260619 18:41:32.934031 18176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "45ba2c5f1f4b453bbe65c4760c3edac5"
format_stamp: "Formatted at 2026-06-19 18:41:32 on dist-test-slave-rxv6"
server_key: "7eadc47e43e7315757508455f23557ba"
server_key_iv: "53a843b36ec2e60d0ea26f5c38eb3507"
server_key_version: "encryptionkey@0"
I20260619 18:41:32.937698 18176 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:41:32.940171 18190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:32.941336 18176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:41:32.941457 18176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "45ba2c5f1f4b453bbe65c4760c3edac5"
format_stamp: "Formatted at 2026-06-19 18:41:32 on dist-test-slave-rxv6"
server_key: "7eadc47e43e7315757508455f23557ba"
server_key_iv: "53a843b36ec2e60d0ea26f5c38eb3507"
server_key_version: "encryptionkey@0"
I20260619 18:41:32.941565 18176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:32.952711 18176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:32.953351 18176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:32.953521 18176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:32.954149 18176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:32.955098 18176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:32.955147 18176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:32.955197 18176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:32.955225 18176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:32.963366 18176 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39535
I20260619 18:41:32.963385 18303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39535 every 8 connection(s)
I20260619 18:41:32.964246 18176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:41:32.967789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 18176
I20260619 18:41:32.967901 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:41:32.968204 392 external_mini_cluster.cc:1428] Setting key 5487ee5469cd1b7d7d7aae7fd81f7d90
I20260619 18:41:32.969961 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35985
--builtin_ntp_servers=127.0.98.20:41377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:41:32.974009 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.966019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39469 (local address 127.0.98.62:35985)
0619 18:41:32.966149 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:32.966153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:32.967274 (+ 1121us) server_negotiation.cc:407] Connection header received
0619 18:41:32.968869 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:32.968873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:32.968927 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:32.969004 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:32.969935 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.970510 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.972002 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.972235 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.972333 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.973521 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:32.973525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:32.973527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:32.973539 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:32.973558 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:32.973567 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:32.973630 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:32.973790 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 18:41:32.973848 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:41:32.975183 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:32.966390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35985 (local address 127.0.98.2:39469)
0619 18:41:32.967094 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:32.967156 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:32.968670 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:32.969199 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:32.969207 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:32.969293 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:32.969775 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:32.969788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.970636 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.970638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:32.971863 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:32.971874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:32.972405 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:32.972408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:32.972591 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:32.973220 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:41:32.973222 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:32.973276 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:41:32.973366 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:32.973649 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:32.973658 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:32.973664 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:41:32.973797 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 18:41:32.974146 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":127,"threads_started":1}
I20260619 18:41:32.976449 18304 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35985
I20260619 18:41:32.976689 18304 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:32.977208 18304 heartbeater.cc:511] Master 127.0.98.62:35985 requested a full tablet report, sending...
I20260619 18:41:32.978291 17985 ts_manager.cc:194] Registered new tserver with Master: 45ba2c5f1f4b453bbe65c4760c3edac5 (127.0.98.2:39535)
I20260619 18:41:32.978899 17985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39469
W20260619 18:41:33.080379 18307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:33.080667 18307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:33.080722 18307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:33.084779 18307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:33.084848 18307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:33.084924 18307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:41:33.089736 18307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:33.090864 18307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:33.092248 18307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:33.101936 18313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:33.101975 18315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:33.102072 18307 server_base.cc:1034] running on GCE node
W20260619 18:41:33.102237 18312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:33.102609 18307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:33.103286 18307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:33.104441 18307 hybrid_clock.cc:648] HybridClock initialized: now 1781894493104423 us; error 32 us; skew 500 ppm
I20260619 18:41:33.106400 18307 webserver.cc:460] Webserver started at http://127.0.98.3:43039/ using document root <none> and password file <none>
I20260619 18:41:33.106942 18307 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:33.107002 18307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:33.107199 18307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:33.109215 18307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "47b3d9376d9f42119733d07dd8fcfbb1"
format_stamp: "Formatted at 2026-06-19 18:41:33 on dist-test-slave-rxv6"
server_key: "4de4129546b2d1abcd427f2c176f0f33"
server_key_iv: "b89493e69530addee4c38c30f24d35e0"
server_key_version: "encryptionkey@0"
I20260619 18:41:33.109743 18307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "47b3d9376d9f42119733d07dd8fcfbb1"
format_stamp: "Formatted at 2026-06-19 18:41:33 on dist-test-slave-rxv6"
server_key: "4de4129546b2d1abcd427f2c176f0f33"
server_key_iv: "b89493e69530addee4c38c30f24d35e0"
server_key_version: "encryptionkey@0"
I20260619 18:41:33.113433 18307 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 18:41:33.115947 18321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:33.117167 18307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 18:41:33.117264 18307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "47b3d9376d9f42119733d07dd8fcfbb1"
format_stamp: "Formatted at 2026-06-19 18:41:33 on dist-test-slave-rxv6"
server_key: "4de4129546b2d1abcd427f2c176f0f33"
server_key_iv: "b89493e69530addee4c38c30f24d35e0"
server_key_version: "encryptionkey@0"
I20260619 18:41:33.117357 18307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:33.144052 18307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:33.144753 18307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:33.144922 18307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:33.145496 18307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:33.146507 18307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:33.146554 18307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:33.146595 18307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:33.146621 18307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:33.154871 18307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46823
I20260619 18:41:33.154901 18434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46823 every 8 connection(s)
I20260619 18:41:33.155792 18307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:41:33.156538 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 18307
I20260619 18:41:33.156670 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:41:33.156982 392 external_mini_cluster.cc:1428] Setting key 67ce38bf6c98fb81e76855063d452519
I20260619 18:41:33.164698 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:33.157815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36465 (local address 127.0.98.62:35985)
0619 18:41:33.157953 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:33.157957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:33.159133 (+ 1176us) server_negotiation.cc:407] Connection header received
0619 18:41:33.160454 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:33.160456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:33.160500 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:33.160562 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:33.161357 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.161889 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.163047 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.163285 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.163366 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:33.164323 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:33.164326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:33.164328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:33.164339 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:33.164358 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:33.164366 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:33.164424 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:33.164536 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 18:41:33.164584 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:41:33.165544 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:33.158257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35985 (local address 127.0.98.3:36465)
0619 18:41:33.158990 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:33.159043 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:33.160283 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:33.160690 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:33.160696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:33.160774 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:33.161248 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:33.161261 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.162012 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.162015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:33.162924 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:33.162933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.163402 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.163405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:33.163541 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:33.164084 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:41:33.164086 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:33.164131 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:41:33.164191 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:33.164431 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:33.164437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:33.164441 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 18:41:33.164532 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 18:41:33.164761 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":113,"threads_started":1}
I20260619 18:41:33.166558 18435 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35985
I20260619 18:41:33.166805 18435 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:33.167318 18435 heartbeater.cc:511] Master 127.0.98.62:35985 requested a full tablet report, sending...
I20260619 18:41:33.168349 17985 ts_manager.cc:194] Registered new tserver with Master: 47b3d9376d9f42119733d07dd8fcfbb1 (127.0.98.3:46823)
I20260619 18:41:33.168900 17985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36465
I20260619 18:41:33.170892 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:41:33.176129 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:33.172763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.0.98.62:35985)
0619 18:41:33.172875 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:33.172879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:33.172984 (+ 105us) server_negotiation.cc:407] Connection header received
0619 18:41:33.173078 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:33.173080 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:33.173114 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:33.173182 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:33.173366 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.173899 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.174784 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.174997 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.175081 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:33.175816 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:33.175820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:33.175822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:33.175832 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:33.175849 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:33.175857 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:33.175912 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:33.175967 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 18:41:33.176013 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 18:41:33.178838 17985 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:41:33.181780 17985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:41:33.195813 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:33.190635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.0.98.2:39535)
0619 18:41:33.191176 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:33.191188 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:33.191216 (+ 28us) server_negotiation.cc:407] Connection header received
0619 18:41:33.191295 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:33.191301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:33.191510 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:33.192053 (+ 543us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:33.192190 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.192929 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.194478 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.195123 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.195229 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:33.195440 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:33.195539 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 18:41:33.195653 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":91,"threads_started":1}
I20260619 18:41:33.195804 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:33.189945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39535 (local address 127.0.0.1:46790)
0619 18:41:33.190489 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:33.190538 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:33.190666 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:33.191718 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:33.191724 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:33.191748 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:33.191897 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:33.191904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.193038 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.193042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:33.194342 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:33.194353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.195241 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.195245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:33.195365 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:33.195404 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 18:41:33.195510 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 18:41:33.195636 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":25,"thread_start_us":97,"threads_started":1}
I20260619 18:41:33.196022 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:33.190137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46823 (local address 127.0.0.1:37728)
0619 18:41:33.190943 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:33.190954 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:33.191038 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:33.191949 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:33.191952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:33.191970 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:33.192063 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:33.192069 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.193484 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.193487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:33.194681 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:33.194691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.195657 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.195660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:33.195747 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:33.195768 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:41:33.195817 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 18:41:33.195887 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":69,"thread_start_us":88,"threads_started":1}
I20260619 18:41:33.196178 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:33.190692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41047 (local address 127.0.0.1:33864)
0619 18:41:33.191201 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:33.191213 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:33.191289 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:33.191928 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:33.191932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:33.191949 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:33.192067 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:33.192074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.193513 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.193516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:33.194582 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:33.194593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.195831 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.195835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:33.195915 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:33.195933 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 18:41:33.195993 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 18:41:33.196047 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":39,"spinlock_wait_cycles":4992,"thread_start_us":73,"threads_started":1}
I20260619 18:41:33.196593 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:33.190878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33864 (local address 127.0.98.1:41047)
0619 18:41:33.191198 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:33.191211 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:33.191272 (+ 61us) server_negotiation.cc:407] Connection header received
0619 18:41:33.191419 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:33.191424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:33.191629 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:33.191778 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:33.192203 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.193366 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.194699 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.195485 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.195591 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:33.195777 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:33.196344 (+ 567us) server_negotiation.cc:299] Negotiation successful
0619 18:41:33.196439 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":92,"threads_started":1}
I20260619 18:41:33.196873 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:33.190181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37728 (local address 127.0.98.3:46823)
0619 18:41:33.191534 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:33.191544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:33.191564 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:41:33.191658 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:33.191663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:33.191823 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:33.191959 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:33.192172 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.193366 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.194792 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.195517 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.196409 (+ 892us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:33.196595 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:33.196654 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 18:41:33.196746 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1257,"thread_start_us":146,"threads_started":1}
I20260619 18:41:33.198974 18238 tablet_service.cc:1467] Processing CreateTablet for tablet 6cdf973367ed4baba2ff0a10f9bc9bb0 (DEFAULT_TABLE table=test-table [id=cbeb2858750f4c6a882e70398aede418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:33.199299 18107 tablet_service.cc:1467] Processing CreateTablet for tablet 6cdf973367ed4baba2ff0a10f9bc9bb0 (DEFAULT_TABLE table=test-table [id=cbeb2858750f4c6a882e70398aede418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:33.199299 18369 tablet_service.cc:1467] Processing CreateTablet for tablet 6cdf973367ed4baba2ff0a10f9bc9bb0 (DEFAULT_TABLE table=test-table [id=cbeb2858750f4c6a882e70398aede418]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:33.199926 18238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cdf973367ed4baba2ff0a10f9bc9bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:33.200285 18107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cdf973367ed4baba2ff0a10f9bc9bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:33.200285 18369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cdf973367ed4baba2ff0a10f9bc9bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:33.206357 18452 tablet_bootstrap.cc:492] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5: Bootstrap starting.
I20260619 18:41:33.207098 18453 tablet_bootstrap.cc:492] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 071789f4ee9941cab13c1b2b74dc9f75: Bootstrap starting.
I20260619 18:41:33.208317 18454 tablet_bootstrap.cc:492] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1: Bootstrap starting.
I20260619 18:41:33.208642 18452 tablet_bootstrap.cc:654] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:33.209189 18453 tablet_bootstrap.cc:654] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 071789f4ee9941cab13c1b2b74dc9f75: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:33.209513 18452 log.cc:824] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:33.209975 18453 log.cc:824] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 071789f4ee9941cab13c1b2b74dc9f75: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:33.210013 18454 tablet_bootstrap.cc:654] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:33.210696 18454 log.cc:824] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:33.211462 18452 tablet_bootstrap.cc:492] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5: No bootstrap required, opened a new log
I20260619 18:41:33.211675 18452 ts_tablet_manager.cc:1397] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:41:33.212514 18454 tablet_bootstrap.cc:492] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1: No bootstrap required, opened a new log
I20260619 18:41:33.212692 18454 ts_tablet_manager.cc:1397] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 18:41:33.215305 18453 tablet_bootstrap.cc:492] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 071789f4ee9941cab13c1b2b74dc9f75: No bootstrap required, opened a new log
I20260619 18:41:33.215474 18453 ts_tablet_manager.cc:1397] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 071789f4ee9941cab13c1b2b74dc9f75: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260619 18:41:33.215902 18452 raft_consensus.cc:348] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ba2c5f1f4b453bbe65c4760c3edac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "47b3d9376d9f42119733d07dd8fcfbb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46823 } } peers { permanent_uuid: "071789f4ee9941cab13c1b2b74dc9f75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41047 } }
I20260619 18:41:33.216172 18452 raft_consensus.cc:374] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:33.216228 18452 raft_consensus.cc:729] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45ba2c5f1f4b453bbe65c4760c3edac5, State: Initialized, Role: FOLLOWER
I20260619 18:41:33.216763 18452 consensus_queue.cc:260] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ba2c5f1f4b453bbe65c4760c3edac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "47b3d9376d9f42119733d07dd8fcfbb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46823 } } peers { permanent_uuid: "071789f4ee9941cab13c1b2b74dc9f75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41047 } }
I20260619 18:41:33.216681 18454 raft_consensus.cc:348] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ba2c5f1f4b453bbe65c4760c3edac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "47b3d9376d9f42119733d07dd8fcfbb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46823 } } peers { permanent_uuid: "071789f4ee9941cab13c1b2b74dc9f75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41047 } }
I20260619 18:41:33.216904 18454 raft_consensus.cc:374] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:33.216954 18454 raft_consensus.cc:729] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b3d9376d9f42119733d07dd8fcfbb1, State: Initialized, Role: FOLLOWER
I20260619 18:41:33.217366 18454 consensus_queue.cc:260] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ba2c5f1f4b453bbe65c4760c3edac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "47b3d9376d9f42119733d07dd8fcfbb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46823 } } peers { permanent_uuid: "071789f4ee9941cab13c1b2b74dc9f75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41047 } }
I20260619 18:41:33.217433 18304 heartbeater.cc:503] Master 127.0.98.62:35985 was elected leader, sending a full tablet report...
I20260619 18:41:33.217793 18452 ts_tablet_manager.cc:1428] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260619 18:41:33.218266 18454 ts_tablet_manager.cc:1428] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260619 18:41:33.218374 18435 heartbeater.cc:503] Master 127.0.98.62:35985 was elected leader, sending a full tablet report...
I20260619 18:41:33.219923 18453 raft_consensus.cc:348] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 071789f4ee9941cab13c1b2b74dc9f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ba2c5f1f4b453bbe65c4760c3edac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "47b3d9376d9f42119733d07dd8fcfbb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46823 } } peers { permanent_uuid: "071789f4ee9941cab13c1b2b74dc9f75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41047 } }
I20260619 18:41:33.220177 18453 raft_consensus.cc:374] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 071789f4ee9941cab13c1b2b74dc9f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:33.220291 18453 raft_consensus.cc:729] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 071789f4ee9941cab13c1b2b74dc9f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 071789f4ee9941cab13c1b2b74dc9f75, State: Initialized, Role: FOLLOWER
I20260619 18:41:33.220749 18453 consensus_queue.cc:260] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 071789f4ee9941cab13c1b2b74dc9f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ba2c5f1f4b453bbe65c4760c3edac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "47b3d9376d9f42119733d07dd8fcfbb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46823 } } peers { permanent_uuid: "071789f4ee9941cab13c1b2b74dc9f75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41047 } }
I20260619 18:41:33.221395 18173 heartbeater.cc:503] Master 127.0.98.62:35985 was elected leader, sending a full tablet report...
I20260619 18:41:33.221701 18453 ts_tablet_manager.cc:1428] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 071789f4ee9941cab13c1b2b74dc9f75: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260619 18:41:33.289136 18174 tablet.cc:2369] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 071789f4ee9941cab13c1b2b74dc9f75: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:41:33.406731 18436 tablet.cc:2369] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:41:33.460489 18459 raft_consensus.cc:482] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:33.460706 18459 raft_consensus.cc:504] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ba2c5f1f4b453bbe65c4760c3edac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "47b3d9376d9f42119733d07dd8fcfbb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46823 } } peers { permanent_uuid: "071789f4ee9941cab13c1b2b74dc9f75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41047 } }
I20260619 18:41:33.461964 18459 leader_election.cc:288] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45ba2c5f1f4b453bbe65c4760c3edac5 (127.0.98.2:39535), 071789f4ee9941cab13c1b2b74dc9f75 (127.0.98.1:41047)
W20260619 18:41:33.465294 18305 tablet.cc:2369] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:41:33.465477 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:33.462323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33775 (local address 127.0.98.2:39535)
0619 18:41:33.462464 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:33.462468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:33.462487 (+ 19us) server_negotiation.cc:407] Connection header received
0619 18:41:33.462552 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:33.462556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:33.462605 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:33.462682 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:33.462967 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.463463 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.464574 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.464958 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.465047 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:33.465107 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:33.465297 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 18:41:33.465369 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 18:41:33.465478 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:33.462221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39535 (local address 127.0.98.3:33775)
0619 18:41:33.462351 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:33.462367 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:33.462467 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:33.462708 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:33.462711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:33.462729 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:33.462849 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:33.462855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.463563 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.463566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:33.464449 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:33.464458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.465063 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.465067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:33.465181 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:33.465198 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 18:41:33.465261 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 18:41:33.465331 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 18:41:33.465992 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:33.462500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56597 (local address 127.0.98.1:41047)
0619 18:41:33.462633 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:33.462638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:33.462796 (+ 158us) server_negotiation.cc:407] Connection header received
0619 18:41:33.462918 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:33.462920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:33.462962 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:33.463024 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:33.463288 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.463774 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.464850 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.465296 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.465409 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:33.465468 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:33.465776 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 18:41:33.465852 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:41:33.466142 18258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cdf973367ed4baba2ff0a10f9bc9bb0" candidate_uuid: "47b3d9376d9f42119733d07dd8fcfbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45ba2c5f1f4b453bbe65c4760c3edac5" is_pre_election: true
I20260619 18:41:33.466300 18461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:33.462232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41047 (local address 127.0.98.3:56597)
0619 18:41:33.462712 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:33.462723 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:33.462809 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:33.463043 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:33.463046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:33.463061 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:33.463183 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:33.463188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.463900 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.463903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:33.464722 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:33.464731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.465550 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.465554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:33.465643 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:33.465664 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:41:33.466118 (+ 454us) client_negotiation.cc:240] Negotiation successful
0619 18:41:33.466191 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":56,"thread_start_us":132,"threads_started":1}
I20260619 18:41:33.466449 18258 raft_consensus.cc:2454] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47b3d9376d9f42119733d07dd8fcfbb1 in term 0.
I20260619 18:41:33.466892 18127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cdf973367ed4baba2ff0a10f9bc9bb0" candidate_uuid: "47b3d9376d9f42119733d07dd8fcfbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "071789f4ee9941cab13c1b2b74dc9f75" is_pre_election: true
I20260619 18:41:33.466960 18323 leader_election.cc:302] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45ba2c5f1f4b453bbe65c4760c3edac5, 47b3d9376d9f42119733d07dd8fcfbb1; no voters:
I20260619 18:41:33.467150 18127 raft_consensus.cc:2454] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 071789f4ee9941cab13c1b2b74dc9f75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47b3d9376d9f42119733d07dd8fcfbb1 in term 0.
I20260619 18:41:33.467231 18459 raft_consensus.cc:2781] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:41:33.467331 18459 raft_consensus.cc:482] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:41:33.467383 18459 raft_consensus.cc:3037] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:33.468449 18459 raft_consensus.cc:504] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ba2c5f1f4b453bbe65c4760c3edac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "47b3d9376d9f42119733d07dd8fcfbb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46823 } } peers { permanent_uuid: "071789f4ee9941cab13c1b2b74dc9f75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41047 } }
I20260619 18:41:33.468777 18459 leader_election.cc:288] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [CANDIDATE]: Term 1 election: Requested vote from peers 45ba2c5f1f4b453bbe65c4760c3edac5 (127.0.98.2:39535), 071789f4ee9941cab13c1b2b74dc9f75 (127.0.98.1:41047)
I20260619 18:41:33.469080 18127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cdf973367ed4baba2ff0a10f9bc9bb0" candidate_uuid: "47b3d9376d9f42119733d07dd8fcfbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "071789f4ee9941cab13c1b2b74dc9f75"
I20260619 18:41:33.469089 18258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cdf973367ed4baba2ff0a10f9bc9bb0" candidate_uuid: "47b3d9376d9f42119733d07dd8fcfbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45ba2c5f1f4b453bbe65c4760c3edac5"
I20260619 18:41:33.469184 18258 raft_consensus.cc:3037] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:33.469192 18127 raft_consensus.cc:3037] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 071789f4ee9941cab13c1b2b74dc9f75 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:33.470227 18127 raft_consensus.cc:2454] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 071789f4ee9941cab13c1b2b74dc9f75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b3d9376d9f42119733d07dd8fcfbb1 in term 1.
I20260619 18:41:33.470234 18258 raft_consensus.cc:2454] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b3d9376d9f42119733d07dd8fcfbb1 in term 1.
I20260619 18:41:33.470539 18323 leader_election.cc:302] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45ba2c5f1f4b453bbe65c4760c3edac5, 47b3d9376d9f42119733d07dd8fcfbb1; no voters:
I20260619 18:41:33.470703 18459 raft_consensus.cc:2781] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:33.470938 18459 raft_consensus.cc:686] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [term 1 LEADER]: Becoming Leader. State: Replica: 47b3d9376d9f42119733d07dd8fcfbb1, State: Running, Role: LEADER
I20260619 18:41:33.471274 18459 consensus_queue.cc:237] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ba2c5f1f4b453bbe65c4760c3edac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } } peers { permanent_uuid: "47b3d9376d9f42119733d07dd8fcfbb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46823 } } peers { permanent_uuid: "071789f4ee9941cab13c1b2b74dc9f75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41047 } }
I20260619 18:41:33.474413 17984 catalog_manager.cc:5495] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47b3d9376d9f42119733d07dd8fcfbb1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "47b3d9376d9f42119733d07dd8fcfbb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45ba2c5f1f4b453bbe65c4760c3edac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47b3d9376d9f42119733d07dd8fcfbb1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "071789f4ee9941cab13c1b2b74dc9f75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41047 } health_report { overall_health: UNKNOWN } } }
I20260619 18:41:33.488372 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:33.485158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.0.98.3:46823)
0619 18:41:33.485285 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:33.485289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:33.485303 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:41:33.485395 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:33.485398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:33.485450 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:33.485534 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:41:33.485707 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.486244 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.487035 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:33.487250 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:33.487332 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:33.487566 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:41:33.488021 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:41:33.488110 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:33.488179 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 18:41:33.488228 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:41:33.494277 18258 raft_consensus.cc:1261] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5 [term 1 FOLLOWER]: Refusing update from remote peer 47b3d9376d9f42119733d07dd8fcfbb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:33.494277 18127 raft_consensus.cc:1261] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 071789f4ee9941cab13c1b2b74dc9f75 [term 1 FOLLOWER]: Refusing update from remote peer 47b3d9376d9f42119733d07dd8fcfbb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:33.495044 18462 consensus_queue.cc:1035] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45ba2c5f1f4b453bbe65c4760c3edac5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:33.495236 18459 consensus_queue.cc:1035] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "071789f4ee9941cab13c1b2b74dc9f75" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:33.502734 18467 mvcc.cc:205] Tried to move back new op lower bound from 7298639845344976896 to 7298639845260906496. Current Snapshot: MvccSnapshot[applied={T|T < 7298639845344976896}]
I20260619 18:41:33.510468 17985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:33.510638 17985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:33.512976 17985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e19fe680e63466a85a33abac1617094: Sending DeleteTablet for 3 replicas of tablet 6cdf973367ed4baba2ff0a10f9bc9bb0
I20260619 18:41:33.513741 18238 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cdf973367ed4baba2ff0a10f9bc9bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:33 UTC) from {username='slave'} at 127.0.0.1:46790
I20260619 18:41:33.513901 18369 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cdf973367ed4baba2ff0a10f9bc9bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:33 UTC) from {username='slave'} at 127.0.0.1:37728
I20260619 18:41:33.513855 18107 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cdf973367ed4baba2ff0a10f9bc9bb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:33 UTC) from {username='slave'} at 127.0.0.1:33864
I20260619 18:41:33.514401 18476 tablet_replica.cc:330] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 071789f4ee9941cab13c1b2b74dc9f75: stopping tablet replica
I20260619 18:41:33.514560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 18045
I20260619 18:41:33.516500 18474 tablet_replica.cc:330] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5: stopping tablet replica
I20260619 18:41:33.516822 18474 raft_consensus.cc:2229] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:33.517167 18474 raft_consensus.cc:2258] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:33.518096 18475 tablet_replica.cc:330] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1: stopping tablet replica
I20260619 18:41:33.518388 18475 raft_consensus.cc:2229] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:41:33.518778 18475 raft_consensus.cc:2258] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:33.518986 18474 ts_tablet_manager.cc:1905] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:33.520500 18475 ts_tablet_manager.cc:1905] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:33.521987 18474 ts_tablet_manager.cc:1918] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:33.522156 18474 log.cc:1196] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/6cdf973367ed4baba2ff0a10f9bc9bb0
I20260619 18:41:33.522737 18474 ts_tablet_manager.cc:1939] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 45ba2c5f1f4b453bbe65c4760c3edac5: Deleting consensus metadata
W20260619 18:41:33.522816 17969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41047 (error 108)
I20260619 18:41:33.523020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 18176
I20260619 18:41:33.524070 18475 ts_tablet_manager.cc:1918] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:33.524178 18475 log.cc:1196] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/6cdf973367ed4baba2ff0a10f9bc9bb0
I20260619 18:41:33.524452 18475 ts_tablet_manager.cc:1939] T 6cdf973367ed4baba2ff0a10f9bc9bb0 P 47b3d9376d9f42119733d07dd8fcfbb1: Deleting consensus metadata
I20260619 18:41:33.524852 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:33.524518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41047 (local address 127.0.0.1:33866)
0619 18:41:33.524659 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:33.524750 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 18:41:33.525027 17969 catalog_manager.cc:4584] TS 071789f4ee9941cab13c1b2b74dc9f75 (127.0.98.1:41047): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6cdf973367ed4baba2ff0a10f9bc9bb0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41047: connect: Connection refused (error 111)
I20260619 18:41:33.525678 17969 catalog_manager.cc:4841] TS 47b3d9376d9f42119733d07dd8fcfbb1 (127.0.98.3:46823): tablet 6cdf973367ed4baba2ff0a10f9bc9bb0 (table test-table [id=cbeb2858750f4c6a882e70398aede418]) successfully deleted
I20260619 18:41:33.528712 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 18307
I20260619 18:41:33.529412 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:33.528897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39535 (local address 127.0.0.1:46792)
0619 18:41:33.529278 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:33.529329 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311}
I20260619 18:41:33.533885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 17954
2026-06-19T18:41:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1184 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:33 dist-test-slave-rxv6 krb5kdc[18480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:33 dist-test-slave-rxv6 krb5kdc[18480](info): set up 2 sockets
Jun 19 18:41:33 dist-test-slave-rxv6 krb5kdc[18480](info): commencing operation
krb5kdc: starting...
W20260619 18:41:34.497668 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:34 dist-test-slave-rxv6 krb5kdc[18480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:41:34.612013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35821
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:34.720300 18493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:34.720568 18493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:34.720624 18493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:34.724612 18493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:34.724674 18493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:34.724702 18493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:34.724721 18493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:34.724750 18493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:34.729964 18493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:34.730971 18493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:34.731961 18493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:34.740027 18499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:34.740027 18501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:34.740161 18493 server_base.cc:1034] running on GCE node
W20260619 18:41:34.739996 18498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:34.740801 18493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:34.741788 18493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:34.742939 18493 hybrid_clock.cc:648] HybridClock initialized: now 1781894494742923 us; error 29 us; skew 500 ppm
Jun 19 18:41:34 dist-test-slave-rxv6 krb5kdc[18480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894494, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:34.745482 18493 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:41:34.746358 18493 webserver.cc:460] Webserver started at http://127.0.98.62:46373/ using document root <none> and password file <none>
I20260619 18:41:34.746855 18493 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:34.746907 18493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:34.747062 18493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:34.748997 18493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "3bd5a0b19b6f472eb0925c616ca5d7f5"
format_stamp: "Formatted at 2026-06-19 18:41:34 on dist-test-slave-rxv6"
server_key: "b710ebcf4f6bf222a6a0b854b1268bd7"
server_key_iv: "8d38898cef2b04fc1a23812410da4c8b"
server_key_version: "encryptionkey@0"
I20260619 18:41:34.749598 18493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "3bd5a0b19b6f472eb0925c616ca5d7f5"
format_stamp: "Formatted at 2026-06-19 18:41:34 on dist-test-slave-rxv6"
server_key: "b710ebcf4f6bf222a6a0b854b1268bd7"
server_key_iv: "8d38898cef2b04fc1a23812410da4c8b"
server_key_version: "encryptionkey@0"
I20260619 18:41:34.753327 18493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:41:34.755826 18508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:34.757078 18493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:41:34.757182 18493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "3bd5a0b19b6f472eb0925c616ca5d7f5"
format_stamp: "Formatted at 2026-06-19 18:41:34 on dist-test-slave-rxv6"
server_key: "b710ebcf4f6bf222a6a0b854b1268bd7"
server_key_iv: "8d38898cef2b04fc1a23812410da4c8b"
server_key_version: "encryptionkey@0"
I20260619 18:41:34.757303 18493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:34.782114 18493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:34.785209 18493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:34.785377 18493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:34.792150 18493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35821
I20260619 18:41:34.792173 18560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35821 every 8 connection(s)
I20260619 18:41:34.793149 18493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:34.795902 18561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:34.798157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 18493
I20260619 18:41:34.798269 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:34.798532 392 external_mini_cluster.cc:1428] Setting key 9d3ac1e56541d8088c8a927e9b0ca1fd
I20260619 18:41:34.801633 18561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5: Bootstrap starting.
I20260619 18:41:34.803784 18561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:34.804455 18561 log.cc:824] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:34.806348 18561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5: No bootstrap required, opened a new log
Jun 19 18:41:34 dist-test-slave-rxv6 krb5kdc[18480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:41:34.809578 18561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd5a0b19b6f472eb0925c616ca5d7f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35821 } }
I20260619 18:41:34.809793 18561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:34.809850 18561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd5a0b19b6f472eb0925c616ca5d7f5, State: Initialized, Role: FOLLOWER
I20260619 18:41:34.810285 18561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd5a0b19b6f472eb0925c616ca5d7f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35821 } }
I20260619 18:41:34.810405 18561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:34.810468 18561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:34.810564 18561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:34.811427 18561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd5a0b19b6f472eb0925c616ca5d7f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35821 } }
I20260619 18:41:34.811832 18561 leader_election.cc:302] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bd5a0b19b6f472eb0925c616ca5d7f5; no voters:
I20260619 18:41:34.812109 18561 leader_election.cc:288] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:34.812206 18566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:34.812419 18566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [term 1 LEADER]: Becoming Leader. State: Replica: 3bd5a0b19b6f472eb0925c616ca5d7f5, State: Running, Role: LEADER
I20260619 18:41:34.812631 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:34.799883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.0.98.62:35821)
0619 18:41:34.800296 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:34.800309 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:34.800344 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:41:34.800985 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:34.801004 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:34.801324 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:34.801644 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:34.802449 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:34.803421 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:34.804279 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:34.804770 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:34.804913 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:34.807448 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:34.807452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:34.807465 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:34.807510 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:34.809599 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:34.810053 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:34.810059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:34.810064 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:34.810204 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:34.810516 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:34.810519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:34.810520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:34.810942 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:34.811139 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:34.811409 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 18:41:34.811691 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":119,"threads_started":1}
I20260619 18:41:34.812784 18566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd5a0b19b6f472eb0925c616ca5d7f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35821 } }
I20260619 18:41:34.813342 18561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:34.813854 18567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bd5a0b19b6f472eb0925c616ca5d7f5. Latest consensus state: current_term: 1 leader_uuid: "3bd5a0b19b6f472eb0925c616ca5d7f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd5a0b19b6f472eb0925c616ca5d7f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35821 } } }
I20260619 18:41:34.814007 18567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:34.814635 18568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bd5a0b19b6f472eb0925c616ca5d7f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd5a0b19b6f472eb0925c616ca5d7f5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35821 } } }
I20260619 18:41:34.814723 18568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:34.815658 18571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:34.818240 18571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:34.823614 18571 catalog_manager.cc:1347] Generated new cluster ID: 9c3168e51f954b5ea187667cf569075d
I20260619 18:41:34.823678 18571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:34.831851 18571 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:34.833115 18571 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:34.837889 18571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5: Generated new TSK 0
I20260619 18:41:34.838619 18571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:41:34.870716 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35821
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:34.982544 18587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:34.982817 18587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:34.982867 18587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:34.986951 18587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:34.987015 18587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:34.987082 18587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:41:34.992494 18587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:34.993665 18587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:34.994792 18587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:41:35.004664 18587 server_base.cc:1034] running on GCE node
W20260619 18:41:35.004616 18595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:35.004616 18592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:35.004624 18593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:35.005252 18587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:35.005892 18587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:35.007103 18587 hybrid_clock.cc:648] HybridClock initialized: now 1781894495007084 us; error 30 us; skew 500 ppm
Jun 19 18:41:35 dist-test-slave-rxv6 krb5kdc[18480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:35.009687 18587 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:41:35.010555 18587 webserver.cc:460] Webserver started at http://127.0.98.1:33337/ using document root <none> and password file <none>
I20260619 18:41:35.011117 18587 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:35.011178 18587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:35.011373 18587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:35.013417 18587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "abcaee30767549c69ce9b39a6a3fe5ef"
format_stamp: "Formatted at 2026-06-19 18:41:35 on dist-test-slave-rxv6"
server_key: "b793109ff6e5094b06637aadc5c3db99"
server_key_iv: "ecba6466f4f320337d2b5e5d94ccf3a3"
server_key_version: "encryptionkey@0"
I20260619 18:41:35.014015 18587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "abcaee30767549c69ce9b39a6a3fe5ef"
format_stamp: "Formatted at 2026-06-19 18:41:35 on dist-test-slave-rxv6"
server_key: "b793109ff6e5094b06637aadc5c3db99"
server_key_iv: "ecba6466f4f320337d2b5e5d94ccf3a3"
server_key_version: "encryptionkey@0"
I20260619 18:41:35.017616 18587 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:41:35.020058 18602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:35.021224 18587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:41:35.021328 18587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "abcaee30767549c69ce9b39a6a3fe5ef"
format_stamp: "Formatted at 2026-06-19 18:41:35 on dist-test-slave-rxv6"
server_key: "b793109ff6e5094b06637aadc5c3db99"
server_key_iv: "ecba6466f4f320337d2b5e5d94ccf3a3"
server_key_version: "encryptionkey@0"
I20260619 18:41:35.021426 18587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:35.040400 18587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:35.043457 18587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:35.043635 18587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:35.044229 18587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:35.045070 18587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:35.045118 18587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:35.045158 18587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:35.045187 18587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:35.053644 18587 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42717
I20260619 18:41:35.053683 18715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42717 every 8 connection(s)
I20260619 18:41:35.054677 18587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:41:35.056841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 18587
I20260619 18:41:35.056949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:41:35.057260 392 external_mini_cluster.cc:1428] Setting key 9db93ab5dccf23612c495087efe9f1b3
Jun 19 18:41:35 dist-test-slave-rxv6 krb5kdc[18480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 18:41:35.070597 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.056405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33111 (local address 127.0.98.62:35821)
0619 18:41:35.056574 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:35.056579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:35.057467 (+ 888us) server_negotiation.cc:407] Connection header received
0619 18:41:35.059183 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:35.059185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:35.059231 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:35.059309 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:35.060950 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.061835 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.063461 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.063749 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.063835 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.066904 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:35.066907 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:35.066909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:35.066938 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:35.068385 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:35.068924 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:35.068928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:35.068929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:35.069045 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:35.069350 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:35.069351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:35.069353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:35.069565 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:35.069643 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:35.070267 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 18:41:35.070400 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 18:41:35.071401 18718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.056787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35821 (local address 127.0.98.1:33111)
0619 18:41:35.057296 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:35.057356 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:35.058947 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:35.059501 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:35.059512 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:35.060005 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:35.060774 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.060798 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.062009 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.062012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.063324 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.063337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.063919 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.063923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.064148 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.064795 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:35.064823 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:35.066702 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:35.068502 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:35.068509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:35.068527 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:35.068789 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:35.069135 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:35.069137 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:35.069139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:35.069279 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:35.069651 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:35.069656 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:35.069974 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 18:41:35.070194 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 18:41:35.070528 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":152,"threads_started":1}
I20260619 18:41:35.072688 18716 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35821
I20260619 18:41:35.072937 18716 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:35.073457 18716 heartbeater.cc:511] Master 127.0.98.62:35821 requested a full tablet report, sending...
I20260619 18:41:35.075073 18525 ts_manager.cc:194] Registered new tserver with Master: abcaee30767549c69ce9b39a6a3fe5ef (127.0.98.1:42717)
I20260619 18:41:35.076210 18525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:41:35.084724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35821
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:35.192291 18721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:35.192574 18721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:35.192634 18721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:35.196686 18721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:35.196763 18721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:35.196846 18721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:41:35.202402 18721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:35.203433 18721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:35.204407 18721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:35.214039 18727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:35.214022 18721 server_base.cc:1034] running on GCE node
W20260619 18:41:35.214030 18726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:35.214003 18729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:35.214775 18721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:35.215502 18721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:35.216684 18721 hybrid_clock.cc:648] HybridClock initialized: now 1781894495216663 us; error 37 us; skew 500 ppm
Jun 19 18:41:35 dist-test-slave-rxv6 krb5kdc[18480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:35.219463 18721 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:41:35.220443 18721 webserver.cc:460] Webserver started at http://127.0.98.2:40183/ using document root <none> and password file <none>
I20260619 18:41:35.220983 18721 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:35.221048 18721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:35.221233 18721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:35.223335 18721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "878742e0e58945ed951c533e466f291e"
format_stamp: "Formatted at 2026-06-19 18:41:35 on dist-test-slave-rxv6"
server_key: "61477c1d0bf03dd5df78db471408a20c"
server_key_iv: "63fd4e52c8e4d27c943c04f21a5ad41e"
server_key_version: "encryptionkey@0"
I20260619 18:41:35.223898 18721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "878742e0e58945ed951c533e466f291e"
format_stamp: "Formatted at 2026-06-19 18:41:35 on dist-test-slave-rxv6"
server_key: "61477c1d0bf03dd5df78db471408a20c"
server_key_iv: "63fd4e52c8e4d27c943c04f21a5ad41e"
server_key_version: "encryptionkey@0"
I20260619 18:41:35.227833 18721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 18:41:35.230384 18736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:35.231659 18721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:41:35.231772 18721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "878742e0e58945ed951c533e466f291e"
format_stamp: "Formatted at 2026-06-19 18:41:35 on dist-test-slave-rxv6"
server_key: "61477c1d0bf03dd5df78db471408a20c"
server_key_iv: "63fd4e52c8e4d27c943c04f21a5ad41e"
server_key_version: "encryptionkey@0"
I20260619 18:41:35.231889 18721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:35.253139 18721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:35.256173 18721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:35.256346 18721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:35.256927 18721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:35.257730 18721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:35.257776 18721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:35.257827 18721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:35.257856 18721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:35.266397 18721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32905
I20260619 18:41:35.266415 18849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32905 every 8 connection(s)
I20260619 18:41:35.267313 18721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:41:35.270850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 18721
I20260619 18:41:35.270964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:41:35.271215 392 external_mini_cluster.cc:1428] Setting key 4b6d563721da17fff552f16d3e228826
Jun 19 18:41:35 dist-test-slave-rxv6 krb5kdc[18480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 18:41:35.281855 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.269121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60015 (local address 127.0.98.62:35821)
0619 18:41:35.269283 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:35.269287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:35.270116 (+ 829us) server_negotiation.cc:407] Connection header received
0619 18:41:35.271432 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:35.271437 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:35.271504 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:35.271605 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:35.273184 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.273950 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.275206 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.275498 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.275585 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.278252 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:35.278257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:35.278260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:35.278294 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:35.279821 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:35.280341 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:35.280345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:35.280347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:35.280442 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:35.280767 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:35.280771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:35.280773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:35.281002 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:35.281102 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:35.281604 (+ 502us) server_negotiation.cc:299] Negotiation successful
0619 18:41:35.281718 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 18:41:35.282739 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.269492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35821 (local address 127.0.98.2:60015)
0619 18:41:35.269979 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:35.270021 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:35.271210 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:35.271856 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:35.271880 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:35.272356 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:35.272985 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.273001 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.274115 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.274117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.275082 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.275092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.275625 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.275627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.275771 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.276332 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:35.276349 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:35.278056 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:35.279941 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:35.279946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:35.279960 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:35.280235 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:35.280534 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:35.280537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:35.280538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:35.280682 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:35.281095 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:35.281102 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:35.281373 (+ 271us) client_negotiation.cc:769] Sending connection context
0619 18:41:35.281554 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 18:41:35.281829 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 18:41:35.284042 18850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35821
I20260619 18:41:35.284272 18850 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:35.284742 18850 heartbeater.cc:511] Master 127.0.98.62:35821 requested a full tablet report, sending...
I20260619 18:41:35.285742 18525 ts_manager.cc:194] Registered new tserver with Master: 878742e0e58945ed951c533e466f291e (127.0.98.2:32905)
I20260619 18:41:35.286332 18525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:41:35.297267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35821
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:35.408733 18855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:35.408998 18855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:35.409039 18855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:35.413302 18855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:35.413360 18855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:35.413430 18855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:41:35.418872 18855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:35.419878 18855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:35.420920 18855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:35.430181 18861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:35.430294 18863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:35.430230 18855 server_base.cc:1034] running on GCE node
W20260619 18:41:35.430246 18860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:35.430794 18855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:35.431463 18855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:35.432711 18855 hybrid_clock.cc:648] HybridClock initialized: now 1781894495432682 us; error 41 us; skew 500 ppm
Jun 19 18:41:35 dist-test-slave-rxv6 krb5kdc[18480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:35.435472 18855 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:41:35.436430 18855 webserver.cc:460] Webserver started at http://127.0.98.3:41773/ using document root <none> and password file <none>
I20260619 18:41:35.436975 18855 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:35.437041 18855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:35.437229 18855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:35.439496 18855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "9336d92a01eb479ebc4a503554c20e89"
format_stamp: "Formatted at 2026-06-19 18:41:35 on dist-test-slave-rxv6"
server_key: "6ae1fada65665f31f1c88fd9f6da2b86"
server_key_iv: "e5df8850eab51399f9c235cb80313b55"
server_key_version: "encryptionkey@0"
I20260619 18:41:35.440047 18855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9336d92a01eb479ebc4a503554c20e89"
format_stamp: "Formatted at 2026-06-19 18:41:35 on dist-test-slave-rxv6"
server_key: "6ae1fada65665f31f1c88fd9f6da2b86"
server_key_iv: "e5df8850eab51399f9c235cb80313b55"
server_key_version: "encryptionkey@0"
I20260619 18:41:35.443964 18855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:41:35.446485 18870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:35.447887 18855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:41:35.448060 18855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "9336d92a01eb479ebc4a503554c20e89"
format_stamp: "Formatted at 2026-06-19 18:41:35 on dist-test-slave-rxv6"
server_key: "6ae1fada65665f31f1c88fd9f6da2b86"
server_key_iv: "e5df8850eab51399f9c235cb80313b55"
server_key_version: "encryptionkey@0"
I20260619 18:41:35.448173 18855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:35.468614 18855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:35.471490 18855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:35.471671 18855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:35.472263 18855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:35.473081 18855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:35.473129 18855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:35.473182 18855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:35.473212 18855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:35.481581 18855 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43469
I20260619 18:41:35.481607 18983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43469 every 8 connection(s)
I20260619 18:41:35.482470 18855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:41:35.483626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 18855
I20260619 18:41:35.483722 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:41:35.484015 392 external_mini_cluster.cc:1428] Setting key 40cbd0f04f4c751bdbe2a5f3dcf001ac
Jun 19 18:41:35 dist-test-slave-rxv6 krb5kdc[18480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:41:35.496882 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.484309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45749 (local address 127.0.98.62:35821)
0619 18:41:35.484470 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:35.484475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:35.485354 (+ 879us) server_negotiation.cc:407] Connection header received
0619 18:41:35.486816 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:35.486818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:35.486862 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:35.486928 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:35.488028 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.488566 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.489625 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.489843 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.489952 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.492348 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:35.492353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:35.492356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:35.492393 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:35.493936 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:35.494706 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:35.494709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:35.494711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:35.494796 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:35.495195 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:35.495200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:35.495201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:35.495441 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:35.495652 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:35.496587 (+ 935us) server_negotiation.cc:299] Negotiation successful
0619 18:41:35.496716 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 18:41:35.498771 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.484740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35821 (local address 127.0.98.3:45749)
0619 18:41:35.485232 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:35.485276 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:35.486625 (+ 1349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:35.487064 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:35.487071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:35.487407 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:35.487884 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.487897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.488661 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.488663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.489513 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.489522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.489988 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.489992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.490146 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.490699 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:35.490715 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:35.492174 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:35.494072 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:35.494079 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:35.494095 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:35.494588 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:35.494898 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:35.494900 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:35.494902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:35.495078 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:35.495733 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:35.495739 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:35.496044 (+ 305us) client_negotiation.cc:769] Sending connection context
0619 18:41:35.497171 (+ 1127us) client_negotiation.cc:240] Negotiation successful
0619 18:41:35.497528 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":137,"threads_started":1}
I20260619 18:41:35.502082 18984 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35821
I20260619 18:41:35.502341 18984 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:35.503115 18984 heartbeater.cc:511] Master 127.0.98.62:35821 requested a full tablet report, sending...
I20260619 18:41:35.504403 18525 ts_manager.cc:194] Registered new tserver with Master: 9336d92a01eb479ebc4a503554c20e89 (127.0.98.3:43469)
I20260619 18:41:35.505162 18525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45749
I20260619 18:41:35.514181 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:41:35.525983 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.516443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57688 (local address 127.0.98.62:35821)
0619 18:41:35.516592 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:35.516597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:35.516859 (+ 262us) server_negotiation.cc:407] Connection header received
0619 18:41:35.517004 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:35.517007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:35.517060 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:35.517141 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:35.518012 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.518640 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.519703 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.519985 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.520091 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.522151 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:35.522156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:35.522159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:35.522189 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:35.523817 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:35.524351 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:35.524355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:35.524357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:35.524467 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:35.524863 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:35.524866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:35.524868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:35.525155 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:35.525296 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:35.525625 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 18:41:35.525780 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 18:41:35.529280 18525 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:41:35.532390 18525 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:41:35.546662 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.541598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.0.98.1:42717)
0619 18:41:35.541957 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:35.541980 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:35.542401 (+ 421us) server_negotiation.cc:407] Connection header received
0619 18:41:35.542482 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:35.542487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:35.542711 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:35.542851 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:35.543222 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.543984 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.545205 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.546015 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.546113 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.546263 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:35.546415 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 18:41:35.546508 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":127,"threads_started":1}
I20260619 18:41:35.546660 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.541443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42717 (local address 127.0.0.1:37536)
0619 18:41:35.542089 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:35.542137 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:35.542270 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:35.542970 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:35.542977 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:35.543001 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:35.543112 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.543117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.544124 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.544127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.545065 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.545076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.546125 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.546129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.546239 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.546275 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:41:35.546377 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 18:41:35.546501 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":1,"spinlock_wait_cycles":1664,"thread_start_us":63,"threads_started":1}
I20260619 18:41:35.548075 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.542143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32905 (local address 127.0.0.1:38068)
0619 18:41:35.542739 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:35.543307 (+ 568us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:35.543395 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:35.543845 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:35.543848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:35.543863 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:35.543971 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.543978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.545626 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.545629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.546475 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.546485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.547713 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.547716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.547808 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.547831 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:41:35.547876 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 18:41:35.547938 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":28,"thread_start_us":93,"threads_started":1}
I20260619 18:41:35.548816 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.543069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38068 (local address 127.0.98.2:32905)
0619 18:41:35.543351 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:35.543362 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:35.543383 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:41:35.543499 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:35.543505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:35.543701 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:35.544103 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:35.544253 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.545488 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.546593 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.547584 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.548333 (+ 749us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.548522 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:35.548581 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 18:41:35.548671 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":99,"threads_started":1}
I20260619 18:41:35.549777 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.541518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43469 (local address 127.0.0.1:38708)
0619 18:41:35.542100 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:35.542137 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:35.542270 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:35.544086 (+ 1816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:35.544089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:35.544110 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:35.544209 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.544215 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.545588 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.545592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.547577 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.547586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.548780 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.548783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.548877 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.549513 (+ 636us) client_negotiation.cc:769] Sending connection context
0619 18:41:35.549581 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 18:41:35.549647 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":1,"spinlock_wait_cycles":5504,"thread_start_us":102,"threads_started":1}
I20260619 18:41:35.549858 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.542630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38708 (local address 127.0.98.3:43469)
0619 18:41:35.543634 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:35.543645 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:35.543665 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:41:35.543733 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:35.543737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:35.543897 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:35.544043 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:35.544317 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.545440 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.547711 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.548495 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.548595 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.548718 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:35.549631 (+ 913us) server_negotiation.cc:299] Negotiation successful
0619 18:41:35.549720 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":134,"threads_started":1}
I20260619 18:41:35.549839 18650 tablet_service.cc:1467] Processing CreateTablet for tablet 55782697348c4e9294fef1aaee13f35e (DEFAULT_TABLE table=test-table [id=d80677db5a804af2aa75a000abb5fcd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:35.550993 18650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55782697348c4e9294fef1aaee13f35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:35.551456 18784 tablet_service.cc:1467] Processing CreateTablet for tablet 55782697348c4e9294fef1aaee13f35e (DEFAULT_TABLE table=test-table [id=d80677db5a804af2aa75a000abb5fcd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:35.551884 18918 tablet_service.cc:1467] Processing CreateTablet for tablet 55782697348c4e9294fef1aaee13f35e (DEFAULT_TABLE table=test-table [id=d80677db5a804af2aa75a000abb5fcd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:35.552476 18784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55782697348c4e9294fef1aaee13f35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:35.552660 18918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55782697348c4e9294fef1aaee13f35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:35.558020 19001 tablet_bootstrap.cc:492] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89: Bootstrap starting.
I20260619 18:41:35.559216 19002 tablet_bootstrap.cc:492] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef: Bootstrap starting.
I20260619 18:41:35.560161 19001 tablet_bootstrap.cc:654] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:35.561151 19002 tablet_bootstrap.cc:654] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:35.561374 19001 log.cc:824] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:35.562261 19002 log.cc:824] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:35.562268 19003 tablet_bootstrap.cc:492] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e: Bootstrap starting.
I20260619 18:41:35.563419 19001 tablet_bootstrap.cc:492] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89: No bootstrap required, opened a new log
I20260619 18:41:35.563609 19001 ts_tablet_manager.cc:1397] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:41:35.563781 19002 tablet_bootstrap.cc:492] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef: No bootstrap required, opened a new log
I20260619 18:41:35.563948 19002 ts_tablet_manager.cc:1397] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:41:35.564174 19003 tablet_bootstrap.cc:654] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:35.565276 19003 log.cc:824] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:35.567085 19002 raft_consensus.cc:348] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcaee30767549c69ce9b39a6a3fe5ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42717 } } peers { permanent_uuid: "9336d92a01eb479ebc4a503554c20e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43469 } } peers { permanent_uuid: "878742e0e58945ed951c533e466f291e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260619 18:41:35.567298 19003 tablet_bootstrap.cc:492] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e: No bootstrap required, opened a new log
I20260619 18:41:35.567292 19002 raft_consensus.cc:374] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:35.567355 19002 raft_consensus.cc:729] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abcaee30767549c69ce9b39a6a3fe5ef, State: Initialized, Role: FOLLOWER
I20260619 18:41:35.567468 19003 ts_tablet_manager.cc:1397] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:41:35.567281 19001 raft_consensus.cc:348] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcaee30767549c69ce9b39a6a3fe5ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42717 } } peers { permanent_uuid: "9336d92a01eb479ebc4a503554c20e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43469 } } peers { permanent_uuid: "878742e0e58945ed951c533e466f291e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260619 18:41:35.567533 19001 raft_consensus.cc:374] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:35.567605 19001 raft_consensus.cc:729] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9336d92a01eb479ebc4a503554c20e89, State: Initialized, Role: FOLLOWER
I20260619 18:41:35.567817 19002 consensus_queue.cc:260] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcaee30767549c69ce9b39a6a3fe5ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42717 } } peers { permanent_uuid: "9336d92a01eb479ebc4a503554c20e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43469 } } peers { permanent_uuid: "878742e0e58945ed951c533e466f291e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260619 18:41:35.568013 19001 consensus_queue.cc:260] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcaee30767549c69ce9b39a6a3fe5ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42717 } } peers { permanent_uuid: "9336d92a01eb479ebc4a503554c20e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43469 } } peers { permanent_uuid: "878742e0e58945ed951c533e466f291e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260619 18:41:35.568450 18716 heartbeater.cc:503] Master 127.0.98.62:35821 was elected leader, sending a full tablet report...
I20260619 18:41:35.568768 19002 ts_tablet_manager.cc:1428] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260619 18:41:35.568877 19001 ts_tablet_manager.cc:1428] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 18:41:35.569191 18984 heartbeater.cc:503] Master 127.0.98.62:35821 was elected leader, sending a full tablet report...
I20260619 18:41:35.570983 19003 raft_consensus.cc:348] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcaee30767549c69ce9b39a6a3fe5ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42717 } } peers { permanent_uuid: "9336d92a01eb479ebc4a503554c20e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43469 } } peers { permanent_uuid: "878742e0e58945ed951c533e466f291e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260619 18:41:35.571259 19003 raft_consensus.cc:374] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:35.571321 19003 raft_consensus.cc:729] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878742e0e58945ed951c533e466f291e, State: Initialized, Role: FOLLOWER
I20260619 18:41:35.571694 19003 consensus_queue.cc:260] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcaee30767549c69ce9b39a6a3fe5ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42717 } } peers { permanent_uuid: "9336d92a01eb479ebc4a503554c20e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43469 } } peers { permanent_uuid: "878742e0e58945ed951c533e466f291e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260619 18:41:35.572309 18850 heartbeater.cc:503] Master 127.0.98.62:35821 was elected leader, sending a full tablet report...
I20260619 18:41:35.572739 19003 ts_tablet_manager.cc:1428] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 18:41:35.733371 18985 tablet.cc:2369] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:41:35.768445 18851 tablet.cc:2369] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 18:41:35.806044 18717 tablet.cc:2369] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:41:35.813484 19007 raft_consensus.cc:482] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:35.813691 19007 raft_consensus.cc:504] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcaee30767549c69ce9b39a6a3fe5ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42717 } } peers { permanent_uuid: "9336d92a01eb479ebc4a503554c20e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43469 } } peers { permanent_uuid: "878742e0e58945ed951c533e466f291e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260619 18:41:35.814934 19007 leader_election.cc:288] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9336d92a01eb479ebc4a503554c20e89 (127.0.98.3:43469), 878742e0e58945ed951c533e466f291e (127.0.98.2:32905)
I20260619 18:41:35.819043 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.815809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60537 (local address 127.0.98.2:32905)
0619 18:41:35.815934 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:35.815938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:35.816018 (+ 80us) server_negotiation.cc:407] Connection header received
0619 18:41:35.816170 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:35.816173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:35.816218 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:35.816272 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:35.816520 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.817034 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.818142 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.818521 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.818601 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.818669 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:35.818837 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 18:41:35.818918 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 18:41:35.819039 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.815626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32905 (local address 127.0.98.1:60537)
0619 18:41:35.815937 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:35.815948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:35.816039 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:35.816296 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:35.816299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:35.816316 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:35.816424 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.816430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.817166 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.817169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.818031 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.818041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.818631 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.818635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.818718 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.818750 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 18:41:35.818811 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 18:41:35.818875 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":15,"thread_start_us":114,"threads_started":1}
I20260619 18:41:35.819092 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.815329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47835 (local address 127.0.98.3:43469)
0619 18:41:35.815456 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:35.815461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:35.815613 (+ 152us) server_negotiation.cc:407] Connection header received
0619 18:41:35.815706 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:35.815709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:35.815758 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:35.815827 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:35.816206 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.816826 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.818126 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.818562 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.818662 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.818724 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:35.818910 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 18:41:35.818985 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 18:41:35.819069 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.815199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43469 (local address 127.0.98.1:47835)
0619 18:41:35.815472 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:35.815486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:35.815611 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:35.815945 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:35.815948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:35.815966 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:35.816085 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.816091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.816929 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.816932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.817999 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.818010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.818689 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.818693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.818789 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.818810 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 18:41:35.818883 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 18:41:35.818946 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":30,"thread_start_us":110,"threads_started":1}
I20260619 18:41:35.819346 19009 raft_consensus.cc:482] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:35.819679 19009 raft_consensus.cc:504] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcaee30767549c69ce9b39a6a3fe5ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42717 } } peers { permanent_uuid: "9336d92a01eb479ebc4a503554c20e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43469 } } peers { permanent_uuid: "878742e0e58945ed951c533e466f291e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260619 18:41:35.819893 18804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55782697348c4e9294fef1aaee13f35e" candidate_uuid: "abcaee30767549c69ce9b39a6a3fe5ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878742e0e58945ed951c533e466f291e" is_pre_election: true
I20260619 18:41:35.819902 18938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55782697348c4e9294fef1aaee13f35e" candidate_uuid: "abcaee30767549c69ce9b39a6a3fe5ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9336d92a01eb479ebc4a503554c20e89" is_pre_election: true
I20260619 18:41:35.820103 18804 raft_consensus.cc:2454] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abcaee30767549c69ce9b39a6a3fe5ef in term 0.
I20260619 18:41:35.820179 18938 raft_consensus.cc:2454] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abcaee30767549c69ce9b39a6a3fe5ef in term 0.
I20260619 18:41:35.820574 18606 leader_election.cc:302] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 878742e0e58945ed951c533e466f291e, abcaee30767549c69ce9b39a6a3fe5ef; no voters:
I20260619 18:41:35.821197 19007 raft_consensus.cc:2781] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:41:35.821290 19007 raft_consensus.cc:482] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:41:35.821343 19007 raft_consensus.cc:3037] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:35.821372 19009 leader_election.cc:288] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abcaee30767549c69ce9b39a6a3fe5ef (127.0.98.1:42717), 9336d92a01eb479ebc4a503554c20e89 (127.0.98.3:43469)
I20260619 18:41:35.822577 19007 raft_consensus.cc:504] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcaee30767549c69ce9b39a6a3fe5ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42717 } } peers { permanent_uuid: "9336d92a01eb479ebc4a503554c20e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43469 } } peers { permanent_uuid: "878742e0e58945ed951c533e466f291e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260619 18:41:35.822909 19007 leader_election.cc:288] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [CANDIDATE]: Term 1 election: Requested vote from peers 9336d92a01eb479ebc4a503554c20e89 (127.0.98.3:43469), 878742e0e58945ed951c533e466f291e (127.0.98.2:32905)
I20260619 18:41:35.823347 18938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55782697348c4e9294fef1aaee13f35e" candidate_uuid: "abcaee30767549c69ce9b39a6a3fe5ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9336d92a01eb479ebc4a503554c20e89"
I20260619 18:41:35.823400 18804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55782697348c4e9294fef1aaee13f35e" candidate_uuid: "abcaee30767549c69ce9b39a6a3fe5ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "878742e0e58945ed951c533e466f291e"
I20260619 18:41:35.823470 18938 raft_consensus.cc:3037] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:35.823508 18804 raft_consensus.cc:3037] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:35.824817 18804 raft_consensus.cc:2454] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abcaee30767549c69ce9b39a6a3fe5ef in term 1.
I20260619 18:41:35.824820 18938 raft_consensus.cc:2454] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abcaee30767549c69ce9b39a6a3fe5ef in term 1.
I20260619 18:41:35.825143 18606 leader_election.cc:302] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 878742e0e58945ed951c533e466f291e, abcaee30767549c69ce9b39a6a3fe5ef; no voters:
I20260619 18:41:35.825318 19007 raft_consensus.cc:2781] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:35.825536 19007 raft_consensus.cc:686] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [term 1 LEADER]: Becoming Leader. State: Replica: abcaee30767549c69ce9b39a6a3fe5ef, State: Running, Role: LEADER
I20260619 18:41:35.825635 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.821106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55633 (local address 127.0.98.1:42717)
0619 18:41:35.821219 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:35.821222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:35.821623 (+ 401us) server_negotiation.cc:407] Connection header received
0619 18:41:35.821740 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:35.821743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:35.821800 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:35.821875 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:35.822144 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.822806 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.824307 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.824864 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.824938 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.825022 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:35.825400 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 18:41:35.825476 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:41:35.826013 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.821048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42717 (local address 127.0.98.2:55633)
0619 18:41:35.821549 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:35.821562 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:35.821659 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:35.821889 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:35.821892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:35.821933 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:35.822040 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.822047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.822919 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.822923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.824169 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.824179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.825136 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.825139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.825236 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.825260 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:41:35.825567 (+ 307us) client_negotiation.cc:240] Negotiation successful
0619 18:41:35.825852 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260619 18:41:35.826181 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.821577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43469 (local address 127.0.98.2:38843)
0619 18:41:35.822183 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:35.822196 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:35.822318 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:35.822626 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:35.822629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:35.822643 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:35.822786 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.822792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.823667 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.823670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.824820 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:35.824831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.825756 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.825759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:35.825845 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.825864 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 18:41:35.825962 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 18:41:35.826040 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":90,"threads_started":1}
I20260619 18:41:35.826200 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.821865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38843 (local address 127.0.98.3:43469)
0619 18:41:35.822026 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:35.822030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:35.822261 (+ 231us) server_negotiation.cc:407] Connection header received
0619 18:41:35.822415 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:35.822418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:35.822460 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:35.822501 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:35.822908 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.823515 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.824953 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.825518 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.825609 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.825664 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:35.826018 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 18:41:35.826095 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 18:41:35.826508 18670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55782697348c4e9294fef1aaee13f35e" candidate_uuid: "878742e0e58945ed951c533e466f291e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abcaee30767549c69ce9b39a6a3fe5ef" is_pre_election: true
I20260619 18:41:35.826583 18938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55782697348c4e9294fef1aaee13f35e" candidate_uuid: "878742e0e58945ed951c533e466f291e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9336d92a01eb479ebc4a503554c20e89" is_pre_election: true
I20260619 18:41:35.826732 18938 raft_consensus.cc:2379] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 878742e0e58945ed951c533e466f291e in current term 1: Already voted for candidate abcaee30767549c69ce9b39a6a3fe5ef in this term.
I20260619 18:41:35.826663 19007 consensus_queue.cc:237] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcaee30767549c69ce9b39a6a3fe5ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42717 } } peers { permanent_uuid: "9336d92a01eb479ebc4a503554c20e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43469 } } peers { permanent_uuid: "878742e0e58945ed951c533e466f291e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } }
I20260619 18:41:35.828408 18740 leader_election.cc:302] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 878742e0e58945ed951c533e466f291e; no voters: 9336d92a01eb479ebc4a503554c20e89, abcaee30767549c69ce9b39a6a3fe5ef
I20260619 18:41:35.828656 19009 raft_consensus.cc:2726] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 18:41:35.830037 18524 catalog_manager.cc:5495] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef reported cstate change: term changed from 0 to 1, leader changed from <none> to abcaee30767549c69ce9b39a6a3fe5ef (127.0.98.1). New cstate: current_term: 1 leader_uuid: "abcaee30767549c69ce9b39a6a3fe5ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abcaee30767549c69ce9b39a6a3fe5ef" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9336d92a01eb479ebc4a503554c20e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "878742e0e58945ed951c533e466f291e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 } health_report { overall_health: UNKNOWN } } }
I20260619 18:41:35.842684 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.839169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.0.98.1:42717)
0619 18:41:35.839301 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:35.839305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:35.839320 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:41:35.839415 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:35.839418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:35.839473 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:35.839572 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:41:35.839847 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.840426 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.841249 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:35.841469 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:35.841567 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:35.841819 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:41:35.842308 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:41:35.842421 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:35.842495 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 18:41:35.842536 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:41:35.848884 18938 raft_consensus.cc:1261] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89 [term 1 FOLLOWER]: Refusing update from remote peer abcaee30767549c69ce9b39a6a3fe5ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:35.848932 18804 raft_consensus.cc:1261] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e [term 1 FOLLOWER]: Refusing update from remote peer abcaee30767549c69ce9b39a6a3fe5ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:35.849568 19014 consensus_queue.cc:1035] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9336d92a01eb479ebc4a503554c20e89" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:35.849769 19007 consensus_queue.cc:1035] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "878742e0e58945ed951c533e466f291e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:35.859025 19018 mvcc.cc:205] Tried to move back new op lower bound from 7298639854988984320 to 7298639854908715008. Current Snapshot: MvccSnapshot[applied={T|T < 7298639854988984320}]
I20260619 18:41:35.859239 19019 mvcc.cc:205] Tried to move back new op lower bound from 7298639854988984320 to 7298639854908715008. Current Snapshot: MvccSnapshot[applied={T|T < 7298639854988984320}]
I20260619 18:41:35.865489 18525 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:35.865697 18525 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:35.868259 18525 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3bd5a0b19b6f472eb0925c616ca5d7f5: Sending DeleteTablet for 3 replicas of tablet 55782697348c4e9294fef1aaee13f35e
I20260619 18:41:35.869285 18650 tablet_service.cc:1514] Processing DeleteTablet for tablet 55782697348c4e9294fef1aaee13f35e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37536
I20260619 18:41:35.869242 18784 tablet_service.cc:1514] Processing DeleteTablet for tablet 55782697348c4e9294fef1aaee13f35e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38068
I20260619 18:41:35.869357 18918 tablet_service.cc:1514] Processing DeleteTablet for tablet 55782697348c4e9294fef1aaee13f35e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38708
I20260619 18:41:35.869881 19028 tablet_replica.cc:330] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89: stopping tablet replica
I20260619 18:41:35.870222 19028 raft_consensus.cc:2229] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:35.870303 19026 tablet_replica.cc:330] T 55782697348c4e9294fef1aaee13f35e P abcaee30767549c69ce9b39a6a3fe5ef: stopping tablet replica
I20260619 18:41:35.870414 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 18587
I20260619 18:41:35.871120 19028 raft_consensus.cc:2258] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:35.872071 19027 tablet_replica.cc:330] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e: stopping tablet replica
I20260619 18:41:35.872365 19027 raft_consensus.cc:2229] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:35.872673 19027 raft_consensus.cc:2258] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:35.874027 19027 ts_tablet_manager.cc:1905] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:35.876657 19027 ts_tablet_manager.cc:1918] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:35.876771 19027 log.cc:1196] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/55782697348c4e9294fef1aaee13f35e
W20260619 18:41:35.876896 18511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42717 (error 108)
I20260619 18:41:35.877080 19027 ts_tablet_manager.cc:1939] T 55782697348c4e9294fef1aaee13f35e P 878742e0e58945ed951c533e466f291e: Deleting consensus metadata
I20260619 18:41:35.877125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 18721
I20260619 18:41:35.878369 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.878105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42717 (local address 127.0.0.1:37548)
0619 18:41:35.878218 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:35.878286 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 18:41:35.878541 18511 catalog_manager.cc:4584] TS abcaee30767549c69ce9b39a6a3fe5ef (127.0.98.1:42717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55782697348c4e9294fef1aaee13f35e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42717: connect: Connection refused (error 111)
I20260619 18:41:35.879493 19028 ts_tablet_manager.cc:1905] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:35.881695 19028 ts_tablet_manager.cc:1918] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:35.881794 19028 log.cc:1196] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/55782697348c4e9294fef1aaee13f35e
I20260619 18:41:35.882117 19028 ts_tablet_manager.cc:1939] T 55782697348c4e9294fef1aaee13f35e P 9336d92a01eb479ebc4a503554c20e89: Deleting consensus metadata
I20260619 18:41:35.883009 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 18855
I20260619 18:41:35.883224 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:35.882922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32905 (local address 127.0.0.1:38082)
0619 18:41:35.883076 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:35.883137 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:32905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 18:41:35.888335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 18493
2026-06-19T18:41:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2355 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T18:41:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:35Z Disabled control of system clock
I20260619 18:41:35.916628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41709
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:41:36.025326 19033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:36.025622 19033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:36.025661 19033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:36.029647 19033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:36.029713 19033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:36.029728 19033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:36.029744 19033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:36.029758 19033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:36.034591 19033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41709
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:36.035650 19033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:36.036991 19033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:36.045487 19039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:36.045522 19041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:36.045487 19038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:36.045970 19033 server_base.cc:1034] running on GCE node
I20260619 18:41:36.046512 19033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:36.047652 19033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:36.048837 19033 hybrid_clock.cc:648] HybridClock initialized: now 1781894496048813 us; error 40 us; skew 500 ppm
I20260619 18:41:36.050802 19033 webserver.cc:460] Webserver started at http://127.0.98.62:38641/ using document root <none> and password file <none>
I20260619 18:41:36.051378 19033 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:36.051445 19033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:36.051659 19033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:36.053762 19033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "eaa85870be39495fbef9d2464ba02d95"
format_stamp: "Formatted at 2026-06-19 18:41:36 on dist-test-slave-rxv6"
server_key: "370292ecb76dacae638caf3394676e13"
server_key_iv: "cd71b2eafed4a1fb500989aca85a609f"
server_key_version: "encryptionkey@0"
I20260619 18:41:36.054368 19033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "eaa85870be39495fbef9d2464ba02d95"
format_stamp: "Formatted at 2026-06-19 18:41:36 on dist-test-slave-rxv6"
server_key: "370292ecb76dacae638caf3394676e13"
server_key_iv: "cd71b2eafed4a1fb500989aca85a609f"
server_key_version: "encryptionkey@0"
I20260619 18:41:36.058115 19033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:41:36.060611 19047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:36.062130 19033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:41:36.062268 19033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "eaa85870be39495fbef9d2464ba02d95"
format_stamp: "Formatted at 2026-06-19 18:41:36 on dist-test-slave-rxv6"
server_key: "370292ecb76dacae638caf3394676e13"
server_key_iv: "cd71b2eafed4a1fb500989aca85a609f"
server_key_version: "encryptionkey@0"
I20260619 18:41:36.062376 19033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:36.083444 19033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:36.084208 19033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:36.084388 19033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:36.091194 19033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41709
I20260619 18:41:36.091220 19099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41709 every 8 connection(s)
I20260619 18:41:36.092173 19033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:36.092459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 19033
I20260619 18:41:36.092571 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:36.092885 392 external_mini_cluster.cc:1428] Setting key 1d28b8c69d47868449a68519be4d4439
I20260619 18:41:36.095860 19100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:36.102659 19100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95: Bootstrap starting.
I20260619 18:41:36.103372 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.094268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42290 (local address 127.0.98.62:41709)
0619 18:41:36.094960 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.094980 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.095025 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:41:36.095893 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.095917 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.096423 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.096800 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:36.097241 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.098730 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.099660 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.100320 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.100482 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.101324 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:36.101331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:36.101347 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:36.101834 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:36.101929 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:36.101956 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:36.102047 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.102194 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.102459 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":123,"threads_started":1}
I20260619 18:41:36.104971 19100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:36.105752 19100 log.cc:824] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:36.107843 19100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95: No bootstrap required, opened a new log
I20260619 18:41:36.111459 19100 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa85870be39495fbef9d2464ba02d95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } }
I20260619 18:41:36.111724 19100 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:36.111775 19100 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa85870be39495fbef9d2464ba02d95, State: Initialized, Role: FOLLOWER
I20260619 18:41:36.112218 19100 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa85870be39495fbef9d2464ba02d95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } }
I20260619 18:41:36.112337 19100 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:36.112396 19100 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:36.112490 19100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:36.113399 19100 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa85870be39495fbef9d2464ba02d95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } }
I20260619 18:41:36.113734 19100 leader_election.cc:302] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaa85870be39495fbef9d2464ba02d95; no voters:
I20260619 18:41:36.114060 19100 leader_election.cc:288] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:36.114192 19105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:36.114408 19105 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [term 1 LEADER]: Becoming Leader. State: Replica: eaa85870be39495fbef9d2464ba02d95, State: Running, Role: LEADER
I20260619 18:41:36.114696 19105 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa85870be39495fbef9d2464ba02d95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } }
I20260619 18:41:36.115259 19100 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:36.116343 19107 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaa85870be39495fbef9d2464ba02d95. Latest consensus state: current_term: 1 leader_uuid: "eaa85870be39495fbef9d2464ba02d95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa85870be39495fbef9d2464ba02d95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } } }
I20260619 18:41:36.116392 19106 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaa85870be39495fbef9d2464ba02d95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa85870be39495fbef9d2464ba02d95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41709 } } }
I20260619 18:41:36.116482 19106 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:36.116482 19107 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:36.116911 19114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:36.120019 19114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:36.125764 19114 catalog_manager.cc:1347] Generated new cluster ID: b7fc16f5d30d433ba02542082c5274a7
I20260619 18:41:36.125838 19114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:36.141503 19114 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:36.142856 19114 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:36.152139 19114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95: Generated new TSK 0
I20260619 18:41:36.153013 19114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:36.157642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41709
--builtin_ntp_servers=127.0.98.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:41:36.270584 19124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:36.270870 19124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:36.270924 19124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:36.275245 19124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:36.275332 19124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:36.275418 19124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:41:36.280593 19124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:36.281749 19124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:36.283197 19124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:36.292762 19129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:36.292776 19130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:36.292750 19132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:36.293012 19124 server_base.cc:1034] running on GCE node
I20260619 18:41:36.293488 19124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:36.294164 19124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:36.295358 19124 hybrid_clock.cc:648] HybridClock initialized: now 1781894496295336 us; error 32 us; skew 500 ppm
I20260619 18:41:36.301926 19124 webserver.cc:460] Webserver started at http://127.0.98.1:34867/ using document root <none> and password file <none>
I20260619 18:41:36.302600 19124 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:36.302677 19124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:36.302893 19124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:36.305053 19124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "a4e8bc1bc3594f2ca5e01c1fce0370a6"
format_stamp: "Formatted at 2026-06-19 18:41:36 on dist-test-slave-rxv6"
server_key: "d9af3ace553f8bdc825cf3002820127d"
server_key_iv: "560940ee17281d29d9250c1a36dfa373"
server_key_version: "encryptionkey@0"
I20260619 18:41:36.305611 19124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a4e8bc1bc3594f2ca5e01c1fce0370a6"
format_stamp: "Formatted at 2026-06-19 18:41:36 on dist-test-slave-rxv6"
server_key: "d9af3ace553f8bdc825cf3002820127d"
server_key_iv: "560940ee17281d29d9250c1a36dfa373"
server_key_version: "encryptionkey@0"
I20260619 18:41:36.309369 19124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:41:36.311883 19138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:36.313158 19124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:41:36.313273 19124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "a4e8bc1bc3594f2ca5e01c1fce0370a6"
format_stamp: "Formatted at 2026-06-19 18:41:36 on dist-test-slave-rxv6"
server_key: "d9af3ace553f8bdc825cf3002820127d"
server_key_iv: "560940ee17281d29d9250c1a36dfa373"
server_key_version: "encryptionkey@0"
I20260619 18:41:36.313375 19124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:36.324038 19124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:36.324756 19124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:36.324942 19124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:36.325538 19124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:36.326538 19124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:36.326596 19124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:36.326655 19124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:36.326679 19124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:36.335224 19124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35109
I20260619 18:41:36.335250 19251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35109 every 8 connection(s)
I20260619 18:41:36.336243 19124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:41:36.344393 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 19124
I20260619 18:41:36.344535 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:41:36.344882 392 external_mini_cluster.cc:1428] Setting key f38510e47f15a1f6a876d92a020a3857
I20260619 18:41:36.346088 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.338047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38123 (local address 127.0.98.62:41709)
0619 18:41:36.338206 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.338210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.339368 (+ 1158us) server_negotiation.cc:407] Connection header received
0619 18:41:36.340715 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.340718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.340769 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.340843 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:36.341930 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.342489 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.343708 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.343942 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.344029 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.345507 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:36.345510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:36.345512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:36.345553 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:36.345575 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:36.345584 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:36.345648 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.345826 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.345944 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:41:36.347115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41709
--builtin_ntp_servers=127.0.98.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:41:36.347110 19254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.338524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.1:38123)
0619 18:41:36.339215 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.339257 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:36.340500 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:36.341003 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:36.341010 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:36.341097 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:36.341745 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.341761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.342671 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.342675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.343574 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.343584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.344232 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.344239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.344457 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.345181 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:41:36.345183 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:36.345240 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:41:36.345326 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:36.345673 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:36.345683 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:36.345689 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:41:36.345809 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 18:41:36.346097 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":111,"threads_started":1}
I20260619 18:41:36.348423 19252 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41709
I20260619 18:41:36.348726 19252 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:36.349382 19252 heartbeater.cc:511] Master 127.0.98.62:41709 requested a full tablet report, sending...
I20260619 18:41:36.351099 19064 ts_manager.cc:194] Registered new tserver with Master: a4e8bc1bc3594f2ca5e01c1fce0370a6 (127.0.98.1:35109)
I20260619 18:41:36.352346 19064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:38123
W20260619 18:41:36.456081 19255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:36.456382 19255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:36.456421 19255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:36.460508 19255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:36.460585 19255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:36.460641 19255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:41:36.465569 19255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:36.466749 19255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:36.468120 19255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:36.477437 19263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:36.477432 19261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:36.477432 19260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:36.477484 19255 server_base.cc:1034] running on GCE node
I20260619 18:41:36.478163 19255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:36.478785 19255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:36.480010 19255 hybrid_clock.cc:648] HybridClock initialized: now 1781894496479978 us; error 42 us; skew 500 ppm
I20260619 18:41:36.481992 19255 webserver.cc:460] Webserver started at http://127.0.98.2:35517/ using document root <none> and password file <none>
I20260619 18:41:36.482596 19255 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:36.482663 19255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:36.482867 19255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:36.484943 19255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "80fd33543e694456bf5accf4509c235b"
format_stamp: "Formatted at 2026-06-19 18:41:36 on dist-test-slave-rxv6"
server_key: "5a573a3b50e749d9d8a6c3bdb82b04e7"
server_key_iv: "8dafcc2abaa0b9809dee3ef7a2236107"
server_key_version: "encryptionkey@0"
I20260619 18:41:36.485498 19255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "80fd33543e694456bf5accf4509c235b"
format_stamp: "Formatted at 2026-06-19 18:41:36 on dist-test-slave-rxv6"
server_key: "5a573a3b50e749d9d8a6c3bdb82b04e7"
server_key_iv: "8dafcc2abaa0b9809dee3ef7a2236107"
server_key_version: "encryptionkey@0"
I20260619 18:41:36.489161 19255 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:41:36.491600 19269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:36.492791 19255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:41:36.492903 19255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "80fd33543e694456bf5accf4509c235b"
format_stamp: "Formatted at 2026-06-19 18:41:36 on dist-test-slave-rxv6"
server_key: "5a573a3b50e749d9d8a6c3bdb82b04e7"
server_key_iv: "8dafcc2abaa0b9809dee3ef7a2236107"
server_key_version: "encryptionkey@0"
I20260619 18:41:36.493011 19255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:36.504029 19255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:36.504694 19255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:36.504859 19255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:36.505406 19255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:36.506356 19255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:36.506403 19255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:36.506455 19255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:36.506480 19255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:36.514361 19255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38557
I20260619 18:41:36.514405 19382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38557 every 8 connection(s)
I20260619 18:41:36.515291 19255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:41:36.523792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 19255
I20260619 18:41:36.523932 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:41:36.524223 392 external_mini_cluster.cc:1428] Setting key 707d10117acd63f3f28ce99792012ecd
I20260619 18:41:36.524981 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.517400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53363 (local address 127.0.98.62:41709)
0619 18:41:36.517560 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.517565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.518672 (+ 1107us) server_negotiation.cc:407] Connection header received
0619 18:41:36.520021 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.520024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.520068 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.520138 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:36.520968 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.521500 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.522794 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.523013 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.523089 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.524419 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:36.524424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:36.524427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:36.524442 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:36.524466 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:36.524476 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:36.524552 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.524763 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.524838 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:41:36.526086 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.517857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.2:53363)
0619 18:41:36.518519 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.518566 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:36.519843 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:36.520278 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:36.520284 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:36.520368 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:36.520853 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.520866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.521627 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.521630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.522670 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.522681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.523209 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.523212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.523368 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.524024 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:41:36.524027 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:36.524073 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:41:36.524141 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:36.524577 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:36.524589 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:36.524595 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:41:36.524760 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 18:41:36.525053 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":105,"threads_started":1}
I20260619 18:41:36.526336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41709
--builtin_ntp_servers=127.0.98.20:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:41:36.528399 19383 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41709
I20260619 18:41:36.528656 19383 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:36.529219 19383 heartbeater.cc:511] Master 127.0.98.62:41709 requested a full tablet report, sending...
I20260619 18:41:36.530454 19064 ts_manager.cc:194] Registered new tserver with Master: 80fd33543e694456bf5accf4509c235b (127.0.98.2:38557)
I20260619 18:41:36.531227 19064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:53363
W20260619 18:41:36.636229 19386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:36.636492 19386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:36.636530 19386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:36.640408 19386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:36.640460 19386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:36.640513 19386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:41:36.645169 19386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:36.646206 19386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:36.647502 19386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:41:36.657245 19386 server_base.cc:1034] running on GCE node
W20260619 18:41:36.657157 19391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:36.657181 19394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:36.657187 19392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:36.657814 19386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:36.658494 19386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:36.659652 19386 hybrid_clock.cc:648] HybridClock initialized: now 1781894496659625 us; error 42 us; skew 500 ppm
I20260619 18:41:36.661468 19386 webserver.cc:460] Webserver started at http://127.0.98.3:35581/ using document root <none> and password file <none>
I20260619 18:41:36.662102 19386 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:36.662173 19386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:36.662367 19386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:36.664383 19386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "a006ea00464648c4a759427158cf58cc"
format_stamp: "Formatted at 2026-06-19 18:41:36 on dist-test-slave-rxv6"
server_key: "20427e175937e697c935b54f4d19b8a5"
server_key_iv: "abd3bebd30cf0280fffbcab2672e33b4"
server_key_version: "encryptionkey@0"
I20260619 18:41:36.664929 19386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a006ea00464648c4a759427158cf58cc"
format_stamp: "Formatted at 2026-06-19 18:41:36 on dist-test-slave-rxv6"
server_key: "20427e175937e697c935b54f4d19b8a5"
server_key_iv: "abd3bebd30cf0280fffbcab2672e33b4"
server_key_version: "encryptionkey@0"
I20260619 18:41:36.668589 19386 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:41:36.671140 19400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:36.672461 19386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:41:36.672576 19386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "a006ea00464648c4a759427158cf58cc"
format_stamp: "Formatted at 2026-06-19 18:41:36 on dist-test-slave-rxv6"
server_key: "20427e175937e697c935b54f4d19b8a5"
server_key_iv: "abd3bebd30cf0280fffbcab2672e33b4"
server_key_version: "encryptionkey@0"
I20260619 18:41:36.672685 19386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:36.689680 19386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:36.690431 19386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:36.690599 19386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:36.691187 19386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:36.692238 19386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:36.692286 19386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:36.692342 19386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:36.692368 19386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:36.701042 19386 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43869
I20260619 18:41:36.701063 19513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43869 every 8 connection(s)
I20260619 18:41:36.701982 19386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:41:36.702478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 19386
I20260619 18:41:36.702587 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:41:36.702873 392 external_mini_cluster.cc:1428] Setting key 0a68543d731dccbde31f9f656733928f
I20260619 18:41:36.710913 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.703829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39933 (local address 127.0.98.62:41709)
0619 18:41:36.703951 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.703954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.705094 (+ 1140us) server_negotiation.cc:407] Connection header received
0619 18:41:36.706574 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.706578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.706621 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.706683 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:36.707564 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.708066 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.709150 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.709353 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.709417 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.710530 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:36.710533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:36.710535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:36.710547 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:36.710567 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:36.710574 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:36.710623 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.710753 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.710802 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:41:36.711786 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.704287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41709 (local address 127.0.98.3:39933)
0619 18:41:36.704943 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.704984 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:36.706395 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:36.706846 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:36.706852 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:36.706938 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:36.707459 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.707471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.708200 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.708203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.709055 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.709065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.709466 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.709469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.709610 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.710284 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:41:36.710285 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:36.710323 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:41:36.710386 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:36.710640 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:36.710646 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:36.710651 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 18:41:36.710752 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 18:41:36.710980 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":101,"threads_started":1}
I20260619 18:41:36.712849 19514 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41709
I20260619 18:41:36.713052 19514 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:36.713560 19514 heartbeater.cc:511] Master 127.0.98.62:41709 requested a full tablet report, sending...
I20260619 18:41:36.714489 19064 ts_manager.cc:194] Registered new tserver with Master: a006ea00464648c4a759427158cf58cc (127.0.98.3:43869)
I20260619 18:41:36.715018 19064 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39933
I20260619 18:41:36.717667 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:41:36.722958 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.719582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42296 (local address 127.0.98.62:41709)
0619 18:41:36.719691 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.719695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.719845 (+ 150us) server_negotiation.cc:407] Connection header received
0619 18:41:36.719939 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.719941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.719977 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.720040 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:36.720330 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.720810 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.721572 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.721768 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.721828 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.722594 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:36.722598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:36.722600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:36.722611 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:36.722627 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:36.722634 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:36.722724 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.722829 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.722857 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:41:36.725557 19064 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:41:36.728518 19064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:41:36.742426 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.737041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.0.98.2:38557)
0619 18:41:36.737412 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.737423 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.737553 (+ 130us) server_negotiation.cc:407] Connection header received
0619 18:41:36.737697 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.737702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.737891 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.738034 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:36.738459 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.739604 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.740828 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.741584 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.741646 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.741801 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.742172 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.742280 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":105,"threads_started":1}
I20260619 18:41:36.742421 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.736569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38557 (local address 127.0.0.1:38560)
0619 18:41:36.737473 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.737483 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:36.737558 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:36.738224 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:36.738226 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:36.738241 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:36.738344 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.738351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.739727 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.739730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.740667 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.740679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.741879 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.741881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.742001 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.742032 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 18:41:36.742126 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 18:41:36.742263 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260619 18:41:36.742951 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.736353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43869 (local address 127.0.0.1:47200)
0619 18:41:36.737120 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.737131 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:36.737270 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:36.738791 (+ 1521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:36.738793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:36.738807 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:36.738864 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.738868 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.739780 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.739783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.741169 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.741179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.742473 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.742477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.742681 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.742703 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 18:41:36.742767 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 18:41:36.742835 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":132,"spinlock_wait_cycles":2816,"thread_start_us":105,"threads_started":1}
I20260619 18:41:36.743446 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.736294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35109 (local address 127.0.0.1:58790)
0619 18:41:36.737058 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.737108 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:36.737270 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:36.738067 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:36.738073 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:36.738095 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:36.738237 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.738243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.739766 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.739769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.741163 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.741179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.742473 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.742477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.743162 (+ 685us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.743186 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:41:36.743256 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 18:41:36.743322 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":705,"spinlock_wait_cycles":33280,"thread_start_us":71,"threads_started":1}
I20260619 18:41:36.743556 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.736402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.0.98.1:35109)
0619 18:41:36.736835 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.736852 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.737222 (+ 370us) server_negotiation.cc:407] Connection header received
0619 18:41:36.737471 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.737477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.737699 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.737829 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:36.738361 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.739604 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.741310 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.742130 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.742236 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.742408 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.743322 (+ 914us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.743416 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":116,"threads_started":1}
I20260619 18:41:36.743934 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.737400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47200 (local address 127.0.98.3:43869)
0619 18:41:36.737753 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.737764 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.737784 (+ 20us) server_negotiation.cc:407] Connection header received
0619 18:41:36.737847 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.737851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.738057 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.738669 (+ 612us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:36.738970 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.739695 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.741310 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.742130 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.742236 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.742408 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.743679 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.743784 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":150,"threads_started":1}
I20260619 18:41:36.745546 19317 tablet_service.cc:1467] Processing CreateTablet for tablet e901ff9062594b90b864e30c2374f7eb (DEFAULT_TABLE table=test-table [id=0335ff71cb3240c0bb68e7ae077586e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:36.745998 19186 tablet_service.cc:1467] Processing CreateTablet for tablet e901ff9062594b90b864e30c2374f7eb (DEFAULT_TABLE table=test-table [id=0335ff71cb3240c0bb68e7ae077586e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:36.746606 19317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e901ff9062594b90b864e30c2374f7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:36.746882 19186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e901ff9062594b90b864e30c2374f7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:36.752893 19532 tablet_bootstrap.cc:492] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b: Bootstrap starting.
I20260619 18:41:36.753734 19531 tablet_bootstrap.cc:492] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6: Bootstrap starting.
I20260619 18:41:36.755805 19532 tablet_bootstrap.cc:654] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:36.756043 19531 tablet_bootstrap.cc:654] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:36.755833 19448 tablet_service.cc:1467] Processing CreateTablet for tablet e901ff9062594b90b864e30c2374f7eb (DEFAULT_TABLE table=test-table [id=0335ff71cb3240c0bb68e7ae077586e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:36.756632 19532 log.cc:824] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:36.756834 19531 log.cc:824] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:36.757040 19448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e901ff9062594b90b864e30c2374f7eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:36.759212 19532 tablet_bootstrap.cc:492] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b: No bootstrap required, opened a new log
I20260619 18:41:36.759212 19531 tablet_bootstrap.cc:492] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6: No bootstrap required, opened a new log
I20260619 18:41:36.759426 19531 ts_tablet_manager.cc:1397] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 18:41:36.759425 19532 ts_tablet_manager.cc:1397] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 18:41:36.763435 19531 raft_consensus.cc:348] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e8bc1bc3594f2ca5e01c1fce0370a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35109 } } peers { permanent_uuid: "a006ea00464648c4a759427158cf58cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43869 } } peers { permanent_uuid: "80fd33543e694456bf5accf4509c235b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
I20260619 18:41:36.763435 19532 raft_consensus.cc:348] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e8bc1bc3594f2ca5e01c1fce0370a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35109 } } peers { permanent_uuid: "a006ea00464648c4a759427158cf58cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43869 } } peers { permanent_uuid: "80fd33543e694456bf5accf4509c235b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
I20260619 18:41:36.763725 19531 raft_consensus.cc:374] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:36.763733 19532 raft_consensus.cc:374] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:36.763783 19531 raft_consensus.cc:729] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e8bc1bc3594f2ca5e01c1fce0370a6, State: Initialized, Role: FOLLOWER
I20260619 18:41:36.763790 19532 raft_consensus.cc:729] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fd33543e694456bf5accf4509c235b, State: Initialized, Role: FOLLOWER
I20260619 18:41:36.764098 19535 tablet_bootstrap.cc:492] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc: Bootstrap starting.
I20260619 18:41:36.764259 19531 consensus_queue.cc:260] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e8bc1bc3594f2ca5e01c1fce0370a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35109 } } peers { permanent_uuid: "a006ea00464648c4a759427158cf58cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43869 } } peers { permanent_uuid: "80fd33543e694456bf5accf4509c235b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
I20260619 18:41:36.764454 19532 consensus_queue.cc:260] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e8bc1bc3594f2ca5e01c1fce0370a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35109 } } peers { permanent_uuid: "a006ea00464648c4a759427158cf58cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43869 } } peers { permanent_uuid: "80fd33543e694456bf5accf4509c235b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
I20260619 18:41:36.764948 19252 heartbeater.cc:503] Master 127.0.98.62:41709 was elected leader, sending a full tablet report...
I20260619 18:41:36.765326 19531 ts_tablet_manager.cc:1428] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260619 18:41:36.766010 19532 ts_tablet_manager.cc:1428] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260619 18:41:36.766167 19384 tablet.cc:2369] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:41:36.766371 19383 heartbeater.cc:503] Master 127.0.98.62:41709 was elected leader, sending a full tablet report...
I20260619 18:41:36.767251 19535 tablet_bootstrap.cc:654] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:36.768178 19535 log.cc:824] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:36.770499 19535 tablet_bootstrap.cc:492] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc: No bootstrap required, opened a new log
I20260619 18:41:36.770701 19535 ts_tablet_manager.cc:1397] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 18:41:36.774034 19535 raft_consensus.cc:348] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e8bc1bc3594f2ca5e01c1fce0370a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35109 } } peers { permanent_uuid: "a006ea00464648c4a759427158cf58cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43869 } } peers { permanent_uuid: "80fd33543e694456bf5accf4509c235b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
I20260619 18:41:36.774286 19535 raft_consensus.cc:374] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:36.774334 19535 raft_consensus.cc:729] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a006ea00464648c4a759427158cf58cc, State: Initialized, Role: FOLLOWER
I20260619 18:41:36.774791 19535 consensus_queue.cc:260] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e8bc1bc3594f2ca5e01c1fce0370a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35109 } } peers { permanent_uuid: "a006ea00464648c4a759427158cf58cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43869 } } peers { permanent_uuid: "80fd33543e694456bf5accf4509c235b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
I20260619 18:41:36.775460 19514 heartbeater.cc:503] Master 127.0.98.62:41709 was elected leader, sending a full tablet report...
I20260619 18:41:36.775712 19535 ts_tablet_manager.cc:1428] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 18:41:36.824231 19539 raft_consensus.cc:482] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:36.824452 19539 raft_consensus.cc:504] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e8bc1bc3594f2ca5e01c1fce0370a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35109 } } peers { permanent_uuid: "a006ea00464648c4a759427158cf58cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43869 } } peers { permanent_uuid: "80fd33543e694456bf5accf4509c235b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
I20260619 18:41:36.825661 19539 leader_election.cc:288] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4e8bc1bc3594f2ca5e01c1fce0370a6 (127.0.98.1:35109), 80fd33543e694456bf5accf4509c235b (127.0.98.2:38557)
I20260619 18:41:36.830281 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.826177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48393 (local address 127.0.98.2:38557)
0619 18:41:36.826346 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.826350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.826425 (+ 75us) server_negotiation.cc:407] Connection header received
0619 18:41:36.826536 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.826538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.826577 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.826642 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:36.826883 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.827570 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.828964 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.829592 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.829699 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.829764 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.830072 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.830145 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 18:41:36.830281 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.826008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51285 (local address 127.0.98.1:35109)
0619 18:41:36.826138 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.826142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.826158 (+ 16us) server_negotiation.cc:407] Connection header received
0619 18:41:36.826215 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.826218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.826293 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.826379 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:36.826669 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.827434 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.828964 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.829592 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.829699 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.829763 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.830073 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.830145 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:41:36.830312 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.825877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35109 (local address 127.0.98.3:51285)
0619 18:41:36.826000 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.826015 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:36.826113 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:36.826409 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:36.826412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:36.826438 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:36.826561 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.826567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.827546 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.827549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.828807 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.828820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.829718 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.829722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.829926 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.829974 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 18:41:36.830053 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 18:41:36.830145 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":73,"spinlock_wait_cycles":384}
I20260619 18:41:36.830328 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.825941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38557 (local address 127.0.98.3:48393)
0619 18:41:36.826342 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.826354 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:36.826447 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:36.826666 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:36.826668 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:36.826690 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:36.826783 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.826787 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.827663 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.827666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.828836 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:36.828843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.829725 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.829728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:36.829925 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.829980 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 18:41:36.830053 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 18:41:36.830145 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":82,"spinlock_wait_cycles":4224,"thread_start_us":124,"threads_started":1}
I20260619 18:41:36.830991 19206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e901ff9062594b90b864e30c2374f7eb" candidate_uuid: "a006ea00464648c4a759427158cf58cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e8bc1bc3594f2ca5e01c1fce0370a6" is_pre_election: true
I20260619 18:41:36.831044 19337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e901ff9062594b90b864e30c2374f7eb" candidate_uuid: "a006ea00464648c4a759427158cf58cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd33543e694456bf5accf4509c235b" is_pre_election: true
I20260619 18:41:36.831346 19206 raft_consensus.cc:2454] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a006ea00464648c4a759427158cf58cc in term 0.
I20260619 18:41:36.831346 19337 raft_consensus.cc:2454] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a006ea00464648c4a759427158cf58cc in term 0.
I20260619 18:41:36.831884 19403 leader_election.cc:302] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a006ea00464648c4a759427158cf58cc, a4e8bc1bc3594f2ca5e01c1fce0370a6; no voters:
I20260619 18:41:36.832176 19539 raft_consensus.cc:2781] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:41:36.832263 19539 raft_consensus.cc:482] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:41:36.832294 19539 raft_consensus.cc:3037] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:36.833279 19539 raft_consensus.cc:504] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e8bc1bc3594f2ca5e01c1fce0370a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35109 } } peers { permanent_uuid: "a006ea00464648c4a759427158cf58cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43869 } } peers { permanent_uuid: "80fd33543e694456bf5accf4509c235b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
I20260619 18:41:36.833614 19539 leader_election.cc:288] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [CANDIDATE]: Term 1 election: Requested vote from peers a4e8bc1bc3594f2ca5e01c1fce0370a6 (127.0.98.1:35109), 80fd33543e694456bf5accf4509c235b (127.0.98.2:38557)
I20260619 18:41:36.833992 19206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e901ff9062594b90b864e30c2374f7eb" candidate_uuid: "a006ea00464648c4a759427158cf58cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4e8bc1bc3594f2ca5e01c1fce0370a6"
I20260619 18:41:36.833992 19337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e901ff9062594b90b864e30c2374f7eb" candidate_uuid: "a006ea00464648c4a759427158cf58cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80fd33543e694456bf5accf4509c235b"
I20260619 18:41:36.834121 19337 raft_consensus.cc:3037] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:36.834121 19206 raft_consensus.cc:3037] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:36.835358 19337 raft_consensus.cc:2454] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a006ea00464648c4a759427158cf58cc in term 1.
I20260619 18:41:36.835358 19206 raft_consensus.cc:2454] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a006ea00464648c4a759427158cf58cc in term 1.
I20260619 18:41:36.835690 19403 leader_election.cc:302] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 80fd33543e694456bf5accf4509c235b, a006ea00464648c4a759427158cf58cc, a4e8bc1bc3594f2ca5e01c1fce0370a6; no voters:
I20260619 18:41:36.835866 19539 raft_consensus.cc:2781] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:36.836094 19539 raft_consensus.cc:686] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [term 1 LEADER]: Becoming Leader. State: Replica: a006ea00464648c4a759427158cf58cc, State: Running, Role: LEADER
I20260619 18:41:36.836434 19539 consensus_queue.cc:237] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e8bc1bc3594f2ca5e01c1fce0370a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35109 } } peers { permanent_uuid: "a006ea00464648c4a759427158cf58cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43869 } } peers { permanent_uuid: "80fd33543e694456bf5accf4509c235b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } }
W20260619 18:41:36.837334 19253 tablet.cc:2369] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:41:36.839694 19064 catalog_manager.cc:5495] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc reported cstate change: term changed from 0 to 1, leader changed from <none> to a006ea00464648c4a759427158cf58cc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a006ea00464648c4a759427158cf58cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4e8bc1bc3594f2ca5e01c1fce0370a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a006ea00464648c4a759427158cf58cc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80fd33543e694456bf5accf4509c235b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 } health_report { overall_health: UNKNOWN } } }
I20260619 18:41:36.867487 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.864081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47212 (local address 127.0.98.3:43869)
0619 18:41:36.864235 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:36.864239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:36.864252 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:41:36.864321 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:36.864323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:36.864375 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:36.864459 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:41:36.864640 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.865123 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.866010 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:36.866230 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:36.866311 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:36.866613 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:41:36.867009 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:41:36.867127 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:36.867312 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 18:41:36.867344 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 18:41:36.873056 19206 raft_consensus.cc:1261] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6 [term 1 FOLLOWER]: Refusing update from remote peer a006ea00464648c4a759427158cf58cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:36.873054 19337 raft_consensus.cc:1261] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b [term 1 FOLLOWER]: Refusing update from remote peer a006ea00464648c4a759427158cf58cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:36.873724 19541 consensus_queue.cc:1035] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4e8bc1bc3594f2ca5e01c1fce0370a6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:36.873935 19539 consensus_queue.cc:1035] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fd33543e694456bf5accf4509c235b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:36.881712 19545 mvcc.cc:205] Tried to move back new op lower bound from 7298639859185295360 to 7298639859044761600. Current Snapshot: MvccSnapshot[applied={T|T < 7298639859185295360}]
I20260619 18:41:36.890108 19062 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:36.890298 19062 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:36.892887 19062 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eaa85870be39495fbef9d2464ba02d95: Sending DeleteTablet for 3 replicas of tablet e901ff9062594b90b864e30c2374f7eb
I20260619 18:41:36.893661 19448 tablet_service.cc:1514] Processing DeleteTablet for tablet e901ff9062594b90b864e30c2374f7eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:36 UTC) from {username='slave'} at 127.0.0.1:47200
I20260619 18:41:36.893658 19186 tablet_service.cc:1514] Processing DeleteTablet for tablet e901ff9062594b90b864e30c2374f7eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:36 UTC) from {username='slave'} at 127.0.0.1:58790
I20260619 18:41:36.893934 19317 tablet_service.cc:1514] Processing DeleteTablet for tablet e901ff9062594b90b864e30c2374f7eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:36 UTC) from {username='slave'} at 127.0.0.1:38560
I20260619 18:41:36.894199 19554 tablet_replica.cc:330] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc: stopping tablet replica
I20260619 18:41:36.894202 19553 tablet_replica.cc:330] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6: stopping tablet replica
I20260619 18:41:36.894500 19554 raft_consensus.cc:2229] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:41:36.894502 19553 raft_consensus.cc:2229] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:36.894821 19553 raft_consensus.cc:2258] T e901ff9062594b90b864e30c2374f7eb P a4e8bc1bc3594f2ca5e01c1fce0370a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:36.894901 19554 raft_consensus.cc:2258] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:36.895747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 19124
I20260619 18:41:36.895291 19555 tablet_replica.cc:330] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b: stopping tablet replica
I20260619 18:41:36.896624 19555 raft_consensus.cc:2229] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:36.896896 19555 raft_consensus.cc:2258] T e901ff9062594b90b864e30c2374f7eb P 80fd33543e694456bf5accf4509c235b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:36.899071 19554 ts_tablet_manager.cc:1905] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:36.901165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 19255
W20260619 18:41:36.901257 19050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35109 (error 108)
I20260619 18:41:36.903472 19554 ts_tablet_manager.cc:1918] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:36.903574 19554 log.cc:1196] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/e901ff9062594b90b864e30c2374f7eb
I20260619 18:41:36.903656 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.903368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35109 (local address 127.0.0.1:58798)
0619 18:41:36.903493 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.903564 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 18:41:36.903868 19554 ts_tablet_manager.cc:1939] T e901ff9062594b90b864e30c2374f7eb P a006ea00464648c4a759427158cf58cc: Deleting consensus metadata
W20260619 18:41:36.903841 19050 catalog_manager.cc:4584] TS a4e8bc1bc3594f2ca5e01c1fce0370a6 (127.0.98.1:35109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e901ff9062594b90b864e30c2374f7eb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35109: connect: Connection refused (error 111)
I20260619 18:41:36.905118 19049 catalog_manager.cc:4841] TS a006ea00464648c4a759427158cf58cc (127.0.98.3:43869): tablet e901ff9062594b90b864e30c2374f7eb (table test-table [id=0335ff71cb3240c0bb68e7ae077586e8]) successfully deleted
I20260619 18:41:36.906844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 19386
I20260619 18:41:36.907558 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:36.906985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38557 (local address 127.0.0.1:38570)
0619 18:41:36.907425 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:36.907476 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363}
I20260619 18:41:36.911616 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 19033
2026-06-19T18:41:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1022 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:36 dist-test-slave-rxv6 krb5kdc[19559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:36 dist-test-slave-rxv6 krb5kdc[19559](info): set up 2 sockets
Jun 19 18:41:36 dist-test-slave-rxv6 krb5kdc[19559](info): commencing operation
krb5kdc: starting...
W20260619 18:41:37.857602 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:37 dist-test-slave-rxv6 krb5kdc[19559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:41:37.971019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41675
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:38.077085 19572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:38.077363 19572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:38.077421 19572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:38.081524 19572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:38.081593 19572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:38.081620 19572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:38.081641 19572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:38.081662 19572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:38.086966 19572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:38.088016 19572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:38.089021 19572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:38.097159 19577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:38.097172 19580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:38.097219 19572 server_base.cc:1034] running on GCE node
W20260619 18:41:38.097141 19578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:38.097841 19572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:38.098955 19572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:38.100118 19572 hybrid_clock.cc:648] HybridClock initialized: now 1781894498100102 us; error 28 us; skew 500 ppm
Jun 19 18:41:38 dist-test-slave-rxv6 krb5kdc[19559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:38.102795 19572 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:41:38.103647 19572 webserver.cc:460] Webserver started at http://127.0.98.62:35645/ using document root <none> and password file <none>
I20260619 18:41:38.104199 19572 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:38.104260 19572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:38.104457 19572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:38.106550 19572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "bf845f6152ce4bb6a96a49a6d6912034"
format_stamp: "Formatted at 2026-06-19 18:41:38 on dist-test-slave-rxv6"
server_key: "b364bea435a19797113111b20c26a7a8"
server_key_iv: "81a228071eb1417726f85cd42f021a8e"
server_key_version: "encryptionkey@0"
I20260619 18:41:38.107080 19572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "bf845f6152ce4bb6a96a49a6d6912034"
format_stamp: "Formatted at 2026-06-19 18:41:38 on dist-test-slave-rxv6"
server_key: "b364bea435a19797113111b20c26a7a8"
server_key_iv: "81a228071eb1417726f85cd42f021a8e"
server_key_version: "encryptionkey@0"
I20260619 18:41:38.111002 19572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:41:38.113616 19587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:38.114948 19572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:41:38.115070 19572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "bf845f6152ce4bb6a96a49a6d6912034"
format_stamp: "Formatted at 2026-06-19 18:41:38 on dist-test-slave-rxv6"
server_key: "b364bea435a19797113111b20c26a7a8"
server_key_iv: "81a228071eb1417726f85cd42f021a8e"
server_key_version: "encryptionkey@0"
I20260619 18:41:38.115177 19572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:38.131090 19572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:38.134080 19572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:38.134258 19572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:38.141048 19572 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41675
I20260619 18:41:38.141064 19639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41675 every 8 connection(s)
I20260619 18:41:38.142022 19572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:38.144753 19640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:38.146955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 19572
I20260619 18:41:38.147082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:38.147395 392 external_mini_cluster.cc:1428] Setting key 994e948e1f8bbdbd3b1b3b98260c8d82
I20260619 18:41:38.151074 19640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034: Bootstrap starting.
I20260619 18:41:38.153690 19640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:38.154594 19640 log.cc:824] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:38.156729 19640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034: No bootstrap required, opened a new log
Jun 19 18:41:38 dist-test-slave-rxv6 krb5kdc[19559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:41:38.160686 19640 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf845f6152ce4bb6a96a49a6d6912034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41675 } }
I20260619 18:41:38.161010 19640 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:38.161072 19640 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf845f6152ce4bb6a96a49a6d6912034, State: Initialized, Role: FOLLOWER
I20260619 18:41:38.161576 19640 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf845f6152ce4bb6a96a49a6d6912034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41675 } }
I20260619 18:41:38.161741 19640 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:38.161861 19640 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:38.162019 19640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:38.163102 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:38.148826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52520 (local address 127.0.98.62:41675)
0619 18:41:38.149249 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:38.149267 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:38.149310 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:41:38.150141 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:38.150165 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:38.150577 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:38.150934 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:38.151625 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.153101 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.154064 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.154743 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.154950 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:38.157572 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:38.157579 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:38.157596 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:38.157628 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:38.160026 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:38.160484 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:38.160490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:38.160496 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:38.160634 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:38.160861 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:38.160864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:38.160866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:38.161372 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:38.161562 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:38.161814 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 18:41:38.162145 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":112,"threads_started":1}
I20260619 18:41:38.163133 19640 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf845f6152ce4bb6a96a49a6d6912034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41675 } }
I20260619 18:41:38.163487 19640 leader_election.cc:302] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf845f6152ce4bb6a96a49a6d6912034; no voters:
I20260619 18:41:38.163822 19640 leader_election.cc:288] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:38.163936 19645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:38.164131 19645 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [term 1 LEADER]: Becoming Leader. State: Replica: bf845f6152ce4bb6a96a49a6d6912034, State: Running, Role: LEADER
I20260619 18:41:38.164480 19645 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf845f6152ce4bb6a96a49a6d6912034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41675 } }
I20260619 18:41:38.164959 19640 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:38.165645 19646 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf845f6152ce4bb6a96a49a6d6912034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf845f6152ce4bb6a96a49a6d6912034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41675 } } }
I20260619 18:41:38.165771 19646 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:38.166042 19647 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf845f6152ce4bb6a96a49a6d6912034. Latest consensus state: current_term: 1 leader_uuid: "bf845f6152ce4bb6a96a49a6d6912034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf845f6152ce4bb6a96a49a6d6912034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41675 } } }
I20260619 18:41:38.166121 19647 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:38.166440 19654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:38.169139 19654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:38.174091 19654 catalog_manager.cc:1347] Generated new cluster ID: 960140b9b3ac4ec3b4f4b7f473d5b370
I20260619 18:41:38.174165 19654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:38.187287 19654 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:38.188524 19654 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:38.196772 19654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034: Generated new TSK 0
I20260619 18:41:38.197518 19654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 18:41:38.230984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41675
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:38.342463 19666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:38.342746 19666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:38.342804 19666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:38.346875 19666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:38.346947 19666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:38.347007 19666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:41:38.352710 19666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:38.353832 19666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:38.354902 19666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:38.364655 19671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:38.364666 19672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:38.364656 19674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:38.364986 19666 server_base.cc:1034] running on GCE node
I20260619 18:41:38.365511 19666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:38.366222 19666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:38.367400 19666 hybrid_clock.cc:648] HybridClock initialized: now 1781894498367376 us; error 40 us; skew 500 ppm
Jun 19 18:41:38 dist-test-slave-rxv6 krb5kdc[19559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:38.370200 19666 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 18:41:38.371137 19666 webserver.cc:460] Webserver started at http://127.0.98.1:45479/ using document root <none> and password file <none>
I20260619 18:41:38.371686 19666 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:38.371750 19666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:38.371951 19666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:38.374128 19666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "6d277b47d23a4c52a3c46ef8df6b5bcd"
format_stamp: "Formatted at 2026-06-19 18:41:38 on dist-test-slave-rxv6"
server_key: "d81b1b7b1af0804465cead528ed6b508"
server_key_iv: "bb7b68a811ec0ecc88a06545df075650"
server_key_version: "encryptionkey@0"
I20260619 18:41:38.374681 19666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6d277b47d23a4c52a3c46ef8df6b5bcd"
format_stamp: "Formatted at 2026-06-19 18:41:38 on dist-test-slave-rxv6"
server_key: "d81b1b7b1af0804465cead528ed6b508"
server_key_iv: "bb7b68a811ec0ecc88a06545df075650"
server_key_version: "encryptionkey@0"
I20260619 18:41:38.378551 19666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:41:38.381104 19681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:38.382568 19666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:41:38.382692 19666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "6d277b47d23a4c52a3c46ef8df6b5bcd"
format_stamp: "Formatted at 2026-06-19 18:41:38 on dist-test-slave-rxv6"
server_key: "d81b1b7b1af0804465cead528ed6b508"
server_key_iv: "bb7b68a811ec0ecc88a06545df075650"
server_key_version: "encryptionkey@0"
I20260619 18:41:38.382804 19666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:38.404793 19666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:38.408100 19666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:38.408306 19666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:38.408923 19666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:38.409770 19666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:38.409818 19666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:38.409873 19666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:38.409936 19666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:38.418442 19666 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39267
I20260619 18:41:38.418475 19794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39267 every 8 connection(s)
I20260619 18:41:38.419380 19666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:41:38.427542 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 19666
I20260619 18:41:38.427672 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:41:38.427953 392 external_mini_cluster.cc:1428] Setting key f231315130daaa6e4fe48778a4fc9f22
Jun 19 18:41:38 dist-test-slave-rxv6 krb5kdc[19559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:41:38.432780 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:38.421270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43283 (local address 127.0.98.62:41675)
0619 18:41:38.421400 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:38.421405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:38.422219 (+ 814us) server_negotiation.cc:407] Connection header received
0619 18:41:38.423131 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:38.423135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:38.423193 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:38.423268 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:38.424468 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.425012 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.426216 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.426451 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.426541 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:38.429132 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:38.429136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:38.429139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:38.429163 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:38.430621 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:38.431237 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:38.431240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:38.431241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:38.431327 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:38.431646 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:38.431650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:38.431652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:38.431846 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:38.431933 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:38.432457 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 18:41:38.432594 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:41:38.433691 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:38.421603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41675 (local address 127.0.98.1:43283)
0619 18:41:38.422077 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:38.422122 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:38.422932 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:38.423458 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:38.423465 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:38.423824 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:38.424317 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:38.424331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.425131 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.425134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:38.426103 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:38.426113 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.426590 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.426594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:38.426756 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:38.427301 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:38.427317 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:38.428973 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:38.430746 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:38.430753 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:38.430769 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:38.431136 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:38.431412 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:38.431416 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:38.431417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:38.431558 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:38.431946 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:38.431953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:38.432206 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 18:41:38.432396 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 18:41:38.432744 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":110,"threads_started":1}
I20260619 18:41:38.435046 19795 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41675
I20260619 18:41:38.435328 19795 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:38.435963 19795 heartbeater.cc:511] Master 127.0.98.62:41675 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 18:41:38.437718 19604 ts_manager.cc:194] Registered new tserver with Master: 6d277b47d23a4c52a3c46ef8df6b5bcd (127.0.98.1:39267)
I20260619 18:41:38.438882 19604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 18:41:38.452663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41675
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:38.562814 19800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:38.563189 19800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:38.563282 19800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:38.567587 19800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:38.567646 19800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:38.567716 19800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:41:38.573050 19800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:38.574079 19800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:38.575062 19800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:41:38.584350 19800 server_base.cc:1034] running on GCE node
W20260619 18:41:38.584303 19805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:38.584326 19806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:38.584303 19808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:38.584897 19800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:38.585565 19800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:38.586745 19800 hybrid_clock.cc:648] HybridClock initialized: now 1781894498586728 us; error 35 us; skew 500 ppm
Jun 19 18:41:38 dist-test-slave-rxv6 krb5kdc[19559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:38.589629 19800 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 18:41:38.590494 19800 webserver.cc:460] Webserver started at http://127.0.98.2:45603/ using document root <none> and password file <none>
I20260619 18:41:38.591018 19800 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:38.591080 19800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:38.591259 19800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:38.593220 19800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "2f3de9f7a0a340bb84ef853f406717ac"
format_stamp: "Formatted at 2026-06-19 18:41:38 on dist-test-slave-rxv6"
server_key: "172ce27d37fa86371a0191c677f737f2"
server_key_iv: "6fbe0e7f800d81e44e9128652cc58424"
server_key_version: "encryptionkey@0"
I20260619 18:41:38.593739 19800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2f3de9f7a0a340bb84ef853f406717ac"
format_stamp: "Formatted at 2026-06-19 18:41:38 on dist-test-slave-rxv6"
server_key: "172ce27d37fa86371a0191c677f737f2"
server_key_iv: "6fbe0e7f800d81e44e9128652cc58424"
server_key_version: "encryptionkey@0"
I20260619 18:41:38.597222 19800 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:41:38.599584 19815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:38.600766 19800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:41:38.600868 19800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "2f3de9f7a0a340bb84ef853f406717ac"
format_stamp: "Formatted at 2026-06-19 18:41:38 on dist-test-slave-rxv6"
server_key: "172ce27d37fa86371a0191c677f737f2"
server_key_iv: "6fbe0e7f800d81e44e9128652cc58424"
server_key_version: "encryptionkey@0"
I20260619 18:41:38.600957 19800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:38.624121 19800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:38.627245 19800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:38.627456 19800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:38.628028 19800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:38.628857 19800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:38.628903 19800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:38.628958 19800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:38.628988 19800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:38.637292 19800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43115
I20260619 18:41:38.637321 19928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43115 every 8 connection(s)
I20260619 18:41:38.638186 19800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:41:38.638720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 19800
I20260619 18:41:38.638854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:41:38.639187 392 external_mini_cluster.cc:1428] Setting key 3d06c8571dd0ac1d302bbbec5ddd1dd8
Jun 19 18:41:38 dist-test-slave-rxv6 krb5kdc[19559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 18:41:38.652318 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:38.640043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46161 (local address 127.0.98.62:41675)
0619 18:41:38.640203 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:38.640208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:38.641190 (+ 982us) server_negotiation.cc:407] Connection header received
0619 18:41:38.642331 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:38.642334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:38.642391 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:38.642473 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:38.643646 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.644175 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.645792 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.646049 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.646129 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:38.648866 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:38.648869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:38.648871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:38.648895 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:38.650154 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:38.650705 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:38.650709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:38.650711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:38.650813 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:38.651217 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:38.651221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:38.651223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:38.651467 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:38.651564 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:38.652013 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 18:41:38.652137 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:41:38.653317 19931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:38.640502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41675 (local address 127.0.98.2:46161)
0619 18:41:38.641019 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:38.641074 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:38.642095 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:38.642633 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:38.642639 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:38.642993 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:38.643504 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:38.643521 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.644290 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.644293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:38.645674 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:38.645685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.646194 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.646199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:38.646402 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:38.646969 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:38.646991 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:38.648633 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:38.650251 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:38.650257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:38.650270 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:38.650564 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:38.650921 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:38.650925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:38.650927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:38.651096 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:38.651569 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:38.651574 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:38.651799 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 18:41:38.651947 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 18:41:38.652193 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":131,"threads_started":1}
I20260619 18:41:38.654520 19929 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41675
I20260619 18:41:38.654785 19929 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:38.655360 19929 heartbeater.cc:511] Master 127.0.98.62:41675 requested a full tablet report, sending...
I20260619 18:41:38.656504 19604 ts_manager.cc:194] Registered new tserver with Master: 2f3de9f7a0a340bb84ef853f406717ac (127.0.98.2:43115)
I20260619 18:41:38.657059 19604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 18:41:38.666755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41675
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:38.776793 19934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:38.777087 19934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:38.777148 19934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:38.781298 19934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:38.781378 19934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:38.781463 19934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:41:38.786993 19934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:38.788157 19934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:38.789178 19934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:38.799026 19940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:38.799165 19934 server_base.cc:1034] running on GCE node
W20260619 18:41:38.799031 19939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:38.799364 19942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:38.799778 19934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:38.800467 19934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:38.801653 19934 hybrid_clock.cc:648] HybridClock initialized: now 1781894498801631 us; error 32 us; skew 500 ppm
Jun 19 18:41:38 dist-test-slave-rxv6 krb5kdc[19559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:38.804411 19934 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 18:41:38.805348 19934 webserver.cc:460] Webserver started at http://127.0.98.3:33889/ using document root <none> and password file <none>
I20260619 18:41:38.805878 19934 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:38.805976 19934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:38.806188 19934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:38.808195 19934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "88e14eff4782474e9c0b1bcf8a3144b1"
format_stamp: "Formatted at 2026-06-19 18:41:38 on dist-test-slave-rxv6"
server_key: "6376b7206e23a24a33201d4b22dc1b31"
server_key_iv: "77ece3ce15422bf25b6ff56d3991b54f"
server_key_version: "encryptionkey@0"
I20260619 18:41:38.808714 19934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "88e14eff4782474e9c0b1bcf8a3144b1"
format_stamp: "Formatted at 2026-06-19 18:41:38 on dist-test-slave-rxv6"
server_key: "6376b7206e23a24a33201d4b22dc1b31"
server_key_iv: "77ece3ce15422bf25b6ff56d3991b54f"
server_key_version: "encryptionkey@0"
I20260619 18:41:38.812433 19934 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:41:38.814764 19949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:38.815914 19934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:41:38.816030 19934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "88e14eff4782474e9c0b1bcf8a3144b1"
format_stamp: "Formatted at 2026-06-19 18:41:38 on dist-test-slave-rxv6"
server_key: "6376b7206e23a24a33201d4b22dc1b31"
server_key_iv: "77ece3ce15422bf25b6ff56d3991b54f"
server_key_version: "encryptionkey@0"
I20260619 18:41:38.816128 19934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:38.842303 19934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:38.845448 19934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:38.845752 19934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:38.846448 19934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:38.847278 19934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:38.847322 19934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:38.847366 19934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:38.847395 19934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:38.856112 19934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41403
I20260619 18:41:38.856134 20062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41403 every 8 connection(s)
I20260619 18:41:38.857010 19934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:41:38.863076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 19934
I20260619 18:41:38.863201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:41:38.863458 392 external_mini_cluster.cc:1428] Setting key 495c9d0a44098860190a376108f6311b
Jun 19 18:41:38 dist-test-slave-rxv6 krb5kdc[19559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894498, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:41:38.870401 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:38.858949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37579 (local address 127.0.98.62:41675)
0619 18:41:38.859099 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:38.859103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:38.859909 (+ 806us) server_negotiation.cc:407] Connection header received
0619 18:41:38.860783 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:38.860787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:38.860843 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:38.860918 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:38.862186 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.862713 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.863922 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.864191 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.864282 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:38.867012 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:38.867015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:38.867017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:38.867042 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:38.868354 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:38.868872 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:38.868875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:38.868876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:38.868962 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:38.869241 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:38.869244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:38.869245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:38.869486 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:38.869572 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:38.870121 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 18:41:38.870230 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 18:41:38.871145 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:38.859308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41675 (local address 127.0.98.3:37579)
0619 18:41:38.859759 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:38.859801 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:38.860577 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:38.861100 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:38.861108 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:38.861526 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:38.862039 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:38.862053 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.862833 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.862836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:38.863786 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:38.863796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.864355 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.864360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:38.864560 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:38.865218 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:41:38.865239 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:38.866848 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:38.868496 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:38.868501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:38.868514 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:38.868757 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:38.869040 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:41:38.869042 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:41:38.869044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:41:38.869166 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:41:38.869578 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:38.869583 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:38.869815 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 18:41:38.870044 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 18:41:38.870314 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":118,"threads_started":1}
I20260619 18:41:38.872319 20063 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41675
I20260619 18:41:38.872593 20063 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:38.873061 20063 heartbeater.cc:511] Master 127.0.98.62:41675 requested a full tablet report, sending...
I20260619 18:41:38.874116 19604 ts_manager.cc:194] Registered new tserver with Master: 88e14eff4782474e9c0b1bcf8a3144b1 (127.0.98.3:41403)
I20260619 18:41:38.874701 19604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37579
I20260619 18:41:38.877465 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:41:38.885880 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:38.879403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.0.98.62:41675)
0619 18:41:38.879521 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:38.879524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:38.879670 (+ 146us) server_negotiation.cc:407] Connection header received
0619 18:41:38.879781 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:38.879783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:38.879821 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:38.879889 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:38.880545 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.881081 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.881962 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.882188 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.882265 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:38.883474 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:38.883477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:38.883479 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:38.883502 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:38.884622 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:38.884951 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:38.884954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:38.884955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:38.885017 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:38.885187 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:38.885189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:38.885190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:38.885386 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:38.885479 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:38.885631 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 18:41:38.885716 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:41:38.888430 19604 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:41:38.891745 19604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:41:38.905453 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:38.899828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39267 (local address 127.0.0.1:33948)
0619 18:41:38.900483 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:38.900497 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:38.900579 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:38.901379 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:38.901385 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:38.901407 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:38.901563 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:38.901571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.902748 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.902751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:38.903982 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:38.904000 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.904944 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.904948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:38.905050 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:38.905086 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 18:41:38.905185 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 18:41:38.905298 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260619 18:41:38.905489 20075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:38.899788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41403 (local address 127.0.0.1:46066)
0619 18:41:38.900161 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:38.900204 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:38.900319 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:38.901556 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:38.901559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:38.901572 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:38.901702 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:38.901710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.903242 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.903246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:38.904166 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:38.904175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.905190 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.905194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:38.905272 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:38.905292 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 18:41:38.905343 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 18:41:38.905398 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":23,"thread_start_us":91,"threads_started":1}
I20260619 18:41:38.905654 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:38.899925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46066 (local address 127.0.98.3:41403)
0619 18:41:38.901159 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:38.901169 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:38.901187 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:41:38.901263 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:38.901267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:38.901425 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:38.901825 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:38.902023 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.903087 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.904301 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.905088 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.905192 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:38.905326 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:38.905378 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 18:41:38.905501 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1140,"thread_start_us":129,"threads_started":1}
I20260619 18:41:38.905821 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:38.900353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43115 (local address 127.0.0.1:40632)
0619 18:41:38.900576 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:38.900588 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:38.900671 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:38.901538 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:38.901542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:38.901560 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:38.901720 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:38.901727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.903180 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.903183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:38.904325 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:38.904334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.905472 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.905476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:38.905559 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:38.905585 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 18:41:38.905631 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 18:41:38.905698 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":25,"spinlock_wait_cycles":1792,"thread_start_us":91,"threads_started":1}
I20260619 18:41:38.906332 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:38.900621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.0.98.1:39267)
0619 18:41:38.900892 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:38.900906 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:38.900927 (+ 21us) server_negotiation.cc:407] Connection header received
0619 18:41:38.901000 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:38.901006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:38.901214 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:38.901714 (+ 500us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:38.901861 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.902630 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.904120 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.904841 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.904934 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:38.905093 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:38.906138 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0619 18:41:38.906221 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":117,"threads_started":1}
I20260619 18:41:38.906416 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:38.900751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40632 (local address 127.0.98.2:43115)
0619 18:41:38.901015 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:38.901026 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:38.901048 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:41:38.901118 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:38.901123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:38.901306 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:38.901690 (+ 384us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:38.901844 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.903041 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.904428 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:38.905346 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:38.906024 (+ 678us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:38.906156 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:38.906201 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 18:41:38.906277 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
I20260619 18:41:38.908648 19863 tablet_service.cc:1467] Processing CreateTablet for tablet de1c18056a5a4bbe8f5db70741efad2d (DEFAULT_TABLE table=test-table [id=96b5672d6dec40bb8a30d0d38c5c8711]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:38.908663 19997 tablet_service.cc:1467] Processing CreateTablet for tablet de1c18056a5a4bbe8f5db70741efad2d (DEFAULT_TABLE table=test-table [id=96b5672d6dec40bb8a30d0d38c5c8711]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:38.909566 19863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1c18056a5a4bbe8f5db70741efad2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:38.909644 19997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1c18056a5a4bbe8f5db70741efad2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:38.909356 19729 tablet_service.cc:1467] Processing CreateTablet for tablet de1c18056a5a4bbe8f5db70741efad2d (DEFAULT_TABLE table=test-table [id=96b5672d6dec40bb8a30d0d38c5c8711]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:38.910269 19729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1c18056a5a4bbe8f5db70741efad2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:38.916098 20080 tablet_bootstrap.cc:492] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac: Bootstrap starting.
I20260619 18:41:38.916864 20081 tablet_bootstrap.cc:492] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1: Bootstrap starting.
I20260619 18:41:38.917318 20082 tablet_bootstrap.cc:492] T de1c18056a5a4bbe8f5db70741efad2d P 6d277b47d23a4c52a3c46ef8df6b5bcd: Bootstrap starting.
I20260619 18:41:38.918450 20080 tablet_bootstrap.cc:654] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:38.918923 20081 tablet_bootstrap.cc:654] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:38.919219 20080 log.cc:824] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:38.919317 20082 tablet_bootstrap.cc:654] T de1c18056a5a4bbe8f5db70741efad2d P 6d277b47d23a4c52a3c46ef8df6b5bcd: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:38.920079 20081 log.cc:824] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:38.920423 20082 log.cc:824] T de1c18056a5a4bbe8f5db70741efad2d P 6d277b47d23a4c52a3c46ef8df6b5bcd: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:38.921600 20080 tablet_bootstrap.cc:492] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac: No bootstrap required, opened a new log
I20260619 18:41:38.921784 20080 ts_tablet_manager.cc:1397] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 18:41:38.922056 20081 tablet_bootstrap.cc:492] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1: No bootstrap required, opened a new log
I20260619 18:41:38.922067 20082 tablet_bootstrap.cc:492] T de1c18056a5a4bbe8f5db70741efad2d P 6d277b47d23a4c52a3c46ef8df6b5bcd: No bootstrap required, opened a new log
I20260619 18:41:38.922235 20081 ts_tablet_manager.cc:1397] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 18:41:38.922235 20082 ts_tablet_manager.cc:1397] T de1c18056a5a4bbe8f5db70741efad2d P 6d277b47d23a4c52a3c46ef8df6b5bcd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:41:38.924602 20080 raft_consensus.cc:348] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e14eff4782474e9c0b1bcf8a3144b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41403 } } peers { permanent_uuid: "6d277b47d23a4c52a3c46ef8df6b5bcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39267 } } peers { permanent_uuid: "2f3de9f7a0a340bb84ef853f406717ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260619 18:41:38.924819 20080 raft_consensus.cc:374] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:38.924870 20080 raft_consensus.cc:729] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3de9f7a0a340bb84ef853f406717ac, State: Initialized, Role: FOLLOWER
I20260619 18:41:38.925272 20080 consensus_queue.cc:260] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e14eff4782474e9c0b1bcf8a3144b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41403 } } peers { permanent_uuid: "6d277b47d23a4c52a3c46ef8df6b5bcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39267 } } peers { permanent_uuid: "2f3de9f7a0a340bb84ef853f406717ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260619 18:41:38.925838 19929 heartbeater.cc:503] Master 127.0.98.62:41675 was elected leader, sending a full tablet report...
I20260619 18:41:38.925850 20082 raft_consensus.cc:348] T de1c18056a5a4bbe8f5db70741efad2d P 6d277b47d23a4c52a3c46ef8df6b5bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e14eff4782474e9c0b1bcf8a3144b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41403 } } peers { permanent_uuid: "6d277b47d23a4c52a3c46ef8df6b5bcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39267 } } peers { permanent_uuid: "2f3de9f7a0a340bb84ef853f406717ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260619 18:41:38.925850 20081 raft_consensus.cc:348] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e14eff4782474e9c0b1bcf8a3144b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41403 } } peers { permanent_uuid: "6d277b47d23a4c52a3c46ef8df6b5bcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39267 } } peers { permanent_uuid: "2f3de9f7a0a340bb84ef853f406717ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260619 18:41:38.926119 20081 raft_consensus.cc:374] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:38.926112 20082 raft_consensus.cc:374] T de1c18056a5a4bbe8f5db70741efad2d P 6d277b47d23a4c52a3c46ef8df6b5bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:38.926172 20081 raft_consensus.cc:729] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88e14eff4782474e9c0b1bcf8a3144b1, State: Initialized, Role: FOLLOWER
I20260619 18:41:38.926173 20082 raft_consensus.cc:729] T de1c18056a5a4bbe8f5db70741efad2d P 6d277b47d23a4c52a3c46ef8df6b5bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d277b47d23a4c52a3c46ef8df6b5bcd, State: Initialized, Role: FOLLOWER
I20260619 18:41:38.926230 20080 ts_tablet_manager.cc:1428] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 18:41:38.926582 20082 consensus_queue.cc:260] T de1c18056a5a4bbe8f5db70741efad2d P 6d277b47d23a4c52a3c46ef8df6b5bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e14eff4782474e9c0b1bcf8a3144b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41403 } } peers { permanent_uuid: "6d277b47d23a4c52a3c46ef8df6b5bcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39267 } } peers { permanent_uuid: "2f3de9f7a0a340bb84ef853f406717ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260619 18:41:38.926748 20081 consensus_queue.cc:260] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e14eff4782474e9c0b1bcf8a3144b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41403 } } peers { permanent_uuid: "6d277b47d23a4c52a3c46ef8df6b5bcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39267 } } peers { permanent_uuid: "2f3de9f7a0a340bb84ef853f406717ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260619 18:41:38.927517 20082 ts_tablet_manager.cc:1428] T de1c18056a5a4bbe8f5db70741efad2d P 6d277b47d23a4c52a3c46ef8df6b5bcd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 18:41:38.927536 20081 ts_tablet_manager.cc:1428] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 18:41:38.927838 19795 heartbeater.cc:503] Master 127.0.98.62:41675 was elected leader, sending a full tablet report...
I20260619 18:41:38.928755 20063 heartbeater.cc:503] Master 127.0.98.62:41675 was elected leader, sending a full tablet report...
I20260619 18:41:39.030972 20088 raft_consensus.cc:482] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:39.031160 20088 raft_consensus.cc:504] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e14eff4782474e9c0b1bcf8a3144b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41403 } } peers { permanent_uuid: "6d277b47d23a4c52a3c46ef8df6b5bcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39267 } } peers { permanent_uuid: "2f3de9f7a0a340bb84ef853f406717ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260619 18:41:39.032264 20088 leader_election.cc:288] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d277b47d23a4c52a3c46ef8df6b5bcd (127.0.98.1:39267), 2f3de9f7a0a340bb84ef853f406717ac (127.0.98.2:43115)
I20260619 18:41:39.035914 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.032612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37403 (local address 127.0.98.1:39267)
0619 18:41:39.032724 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:39.032729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:39.032971 (+ 242us) server_negotiation.cc:407] Connection header received
0619 18:41:39.033086 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:39.033088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:39.033129 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:39.033190 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:39.033440 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.033947 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.034988 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.035374 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.035459 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.035518 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:39.035729 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 18:41:39.035795 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:41:39.035931 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.032553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39267 (local address 127.0.98.3:37403)
0619 18:41:39.032884 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:39.032896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:39.032978 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:39.033224 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:39.033227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:39.033241 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:39.033339 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:39.033343 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.034045 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.034048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:39.034861 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:39.034869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.035479 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.035482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:39.035617 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.035635 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 18:41:39.035696 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 18:41:39.035764 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":37,"thread_start_us":134,"threads_started":1}
I20260619 18:41:39.035948 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.032698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53323 (local address 127.0.98.2:43115)
0619 18:41:39.032818 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:39.032822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:39.032839 (+ 17us) server_negotiation.cc:407] Connection header received
0619 18:41:39.032889 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:39.032892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:39.032932 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:39.033014 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:39.033276 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.033763 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.034941 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.035340 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.035430 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.035494 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:39.035774 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 18:41:39.035844 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:41:39.035948 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.032515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43115 (local address 127.0.98.3:53323)
0619 18:41:39.032708 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:39.032723 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:39.032814 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:39.033044 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:39.033047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:39.033064 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:39.033168 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:39.033174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.033865 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.033868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:39.034816 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:39.034825 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.035451 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.035454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:39.035660 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.035678 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 18:41:39.035745 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 18:41:39.035811 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":148}
I20260619 18:41:39.036763 19749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1c18056a5a4bbe8f5db70741efad2d" candidate_uuid: "88e14eff4782474e9c0b1bcf8a3144b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d277b47d23a4c52a3c46ef8df6b5bcd" is_pre_election: true
I20260619 18:41:39.036764 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1c18056a5a4bbe8f5db70741efad2d" candidate_uuid: "88e14eff4782474e9c0b1bcf8a3144b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3de9f7a0a340bb84ef853f406717ac" is_pre_election: true
I20260619 18:41:39.037081 19749 raft_consensus.cc:2454] T de1c18056a5a4bbe8f5db70741efad2d P 6d277b47d23a4c52a3c46ef8df6b5bcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88e14eff4782474e9c0b1bcf8a3144b1 in term 0.
I20260619 18:41:39.037081 19883 raft_consensus.cc:2454] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88e14eff4782474e9c0b1bcf8a3144b1 in term 0.
I20260619 18:41:39.037575 19950 leader_election.cc:302] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3de9f7a0a340bb84ef853f406717ac, 88e14eff4782474e9c0b1bcf8a3144b1; no voters:
I20260619 18:41:39.037817 20088 raft_consensus.cc:2781] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:41:39.037899 20088 raft_consensus.cc:482] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:41:39.037959 20088 raft_consensus.cc:3037] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:39.038931 20088 raft_consensus.cc:504] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e14eff4782474e9c0b1bcf8a3144b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41403 } } peers { permanent_uuid: "6d277b47d23a4c52a3c46ef8df6b5bcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39267 } } peers { permanent_uuid: "2f3de9f7a0a340bb84ef853f406717ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260619 18:41:39.039256 20088 leader_election.cc:288] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [CANDIDATE]: Term 1 election: Requested vote from peers 6d277b47d23a4c52a3c46ef8df6b5bcd (127.0.98.1:39267), 2f3de9f7a0a340bb84ef853f406717ac (127.0.98.2:43115)
I20260619 18:41:39.039614 19749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1c18056a5a4bbe8f5db70741efad2d" candidate_uuid: "88e14eff4782474e9c0b1bcf8a3144b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d277b47d23a4c52a3c46ef8df6b5bcd"
I20260619 18:41:39.039723 19749 raft_consensus.cc:3037] T de1c18056a5a4bbe8f5db70741efad2d P 6d277b47d23a4c52a3c46ef8df6b5bcd [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:39.039819 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1c18056a5a4bbe8f5db70741efad2d" candidate_uuid: "88e14eff4782474e9c0b1bcf8a3144b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3de9f7a0a340bb84ef853f406717ac"
I20260619 18:41:39.039937 19883 raft_consensus.cc:3037] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:39.040710 19749 raft_consensus.cc:2454] T de1c18056a5a4bbe8f5db70741efad2d P 6d277b47d23a4c52a3c46ef8df6b5bcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e14eff4782474e9c0b1bcf8a3144b1 in term 1.
I20260619 18:41:39.040937 19883 raft_consensus.cc:2454] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88e14eff4782474e9c0b1bcf8a3144b1 in term 1.
I20260619 18:41:39.041015 19952 leader_election.cc:302] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d277b47d23a4c52a3c46ef8df6b5bcd, 88e14eff4782474e9c0b1bcf8a3144b1; no voters:
I20260619 18:41:39.041169 20088 raft_consensus.cc:2781] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:39.041411 20088 raft_consensus.cc:686] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [term 1 LEADER]: Becoming Leader. State: Replica: 88e14eff4782474e9c0b1bcf8a3144b1, State: Running, Role: LEADER
I20260619 18:41:39.041709 20088 consensus_queue.cc:237] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e14eff4782474e9c0b1bcf8a3144b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41403 } } peers { permanent_uuid: "6d277b47d23a4c52a3c46ef8df6b5bcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39267 } } peers { permanent_uuid: "2f3de9f7a0a340bb84ef853f406717ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260619 18:41:39.044664 19603 catalog_manager.cc:5495] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88e14eff4782474e9c0b1bcf8a3144b1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "88e14eff4782474e9c0b1bcf8a3144b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88e14eff4782474e9c0b1bcf8a3144b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d277b47d23a4c52a3c46ef8df6b5bcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3de9f7a0a340bb84ef853f406717ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } health_report { overall_health: UNKNOWN } } }
I20260619 18:41:39.057965 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.054608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46080 (local address 127.0.98.3:41403)
0619 18:41:39.054736 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:39.054741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:39.054754 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:41:39.054838 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:39.054840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:39.054891 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:39.054979 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:41:39.055220 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.055713 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.056532 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.056745 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.056828 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.057126 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:41:39.057546 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:41:39.057652 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:39.057721 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 18:41:39.057779 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:41:39.063452 19883 raft_consensus.cc:1261] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac [term 1 FOLLOWER]: Refusing update from remote peer 88e14eff4782474e9c0b1bcf8a3144b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:39.063474 19749 raft_consensus.cc:1261] T de1c18056a5a4bbe8f5db70741efad2d P 6d277b47d23a4c52a3c46ef8df6b5bcd [term 1 FOLLOWER]: Refusing update from remote peer 88e14eff4782474e9c0b1bcf8a3144b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:39.064108 20088 consensus_queue.cc:1035] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3de9f7a0a340bb84ef853f406717ac" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:39.064284 20090 consensus_queue.cc:1035] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d277b47d23a4c52a3c46ef8df6b5bcd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:39.071820 20093 mvcc.cc:205] Tried to move back new op lower bound from 7298639868157054976 to 7298639868077404160. Current Snapshot: MvccSnapshot[applied={T|T < 7298639868157054976}]
I20260619 18:41:39.073565 20095 mvcc.cc:205] Tried to move back new op lower bound from 7298639868157054976 to 7298639868077404160. Current Snapshot: MvccSnapshot[applied={T|T < 7298639868157054976}]
I20260619 18:41:39.082196 19599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:39.082362 19599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:39.084823 19599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf845f6152ce4bb6a96a49a6d6912034: Sending DeleteTablet for 3 replicas of tablet de1c18056a5a4bbe8f5db70741efad2d
I20260619 18:41:39.085608 19729 tablet_service.cc:1514] Processing DeleteTablet for tablet de1c18056a5a4bbe8f5db70741efad2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33948
I20260619 18:41:39.085611 19997 tablet_service.cc:1514] Processing DeleteTablet for tablet de1c18056a5a4bbe8f5db70741efad2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46066
I20260619 18:41:39.085731 19863 tablet_service.cc:1514] Processing DeleteTablet for tablet de1c18056a5a4bbe8f5db70741efad2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40632
I20260619 18:41:39.086174 20104 tablet_replica.cc:330] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac: stopping tablet replica
I20260619 18:41:39.086510 20102 tablet_replica.cc:330] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1: stopping tablet replica
I20260619 18:41:39.086596 20104 raft_consensus.cc:2229] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:39.086766 20102 raft_consensus.cc:2229] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:41:39.086903 20104 raft_consensus.cc:2258] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:39.086931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 19666
I20260619 18:41:39.087137 20102 raft_consensus.cc:2258] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:39.088492 20102 ts_tablet_manager.cc:1905] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:39.088488 20104 ts_tablet_manager.cc:1905] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:39.091082 20102 ts_tablet_manager.cc:1918] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:39.091082 20104 ts_tablet_manager.cc:1918] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:39.091171 20104 log.cc:1196] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/de1c18056a5a4bbe8f5db70741efad2d
I20260619 18:41:39.091171 20102 log.cc:1196] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/de1c18056a5a4bbe8f5db70741efad2d
I20260619 18:41:39.091471 20104 ts_tablet_manager.cc:1939] T de1c18056a5a4bbe8f5db70741efad2d P 2f3de9f7a0a340bb84ef853f406717ac: Deleting consensus metadata
I20260619 18:41:39.091480 20102 ts_tablet_manager.cc:1939] T de1c18056a5a4bbe8f5db70741efad2d P 88e14eff4782474e9c0b1bcf8a3144b1: Deleting consensus metadata
I20260619 18:41:39.092640 19588 catalog_manager.cc:4841] TS 2f3de9f7a0a340bb84ef853f406717ac (127.0.98.2:43115): tablet de1c18056a5a4bbe8f5db70741efad2d (table test-table [id=96b5672d6dec40bb8a30d0d38c5c8711]) successfully deleted
I20260619 18:41:39.092747 19591 catalog_manager.cc:4841] TS 88e14eff4782474e9c0b1bcf8a3144b1 (127.0.98.3:41403): tablet de1c18056a5a4bbe8f5db70741efad2d (table test-table [id=96b5672d6dec40bb8a30d0d38c5c8711]) successfully deleted
W20260619 18:41:39.096095 19590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39267 (error 108)
I20260619 18:41:39.096854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 19800
I20260619 18:41:39.097126 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.096820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39267 (local address 127.0.0.1:33952)
0619 18:41:39.096960 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:39.097040 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 18:41:39.097591 19590 catalog_manager.cc:4584] TS 6d277b47d23a4c52a3c46ef8df6b5bcd (127.0.98.1:39267): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de1c18056a5a4bbe8f5db70741efad2d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39267: connect: Connection refused (error 111)
I20260619 18:41:39.101675 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 19934
I20260619 18:41:39.106532 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 19572
2026-06-19T18:41:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2194 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T18:41:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:39Z Disabled control of system clock
I20260619 18:41:39.133998 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44611
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:41:39.240541 20109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:39.240813 20109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:39.240851 20109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:39.244837 20109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:39.244906 20109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:39.244921 20109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:39.244936 20109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:39.244949 20109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:39.249727 20109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44611
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:39.250900 20109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:39.252265 20109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:39.260458 20117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:39.260452 20114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:39.260452 20115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:39.260635 20109 server_base.cc:1034] running on GCE node
I20260619 18:41:39.261241 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:39.262352 20109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:39.263520 20109 hybrid_clock.cc:648] HybridClock initialized: now 1781894499263495 us; error 38 us; skew 500 ppm
I20260619 18:41:39.265255 20109 webserver.cc:460] Webserver started at http://127.0.98.62:37761/ using document root <none> and password file <none>
I20260619 18:41:39.265825 20109 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:39.265892 20109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:39.266101 20109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:39.268055 20109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "7748f42d7f1f4a198af8eda49775b80f"
format_stamp: "Formatted at 2026-06-19 18:41:39 on dist-test-slave-rxv6"
server_key: "b5cde50176cc625a9be4cecd7a9f081f"
server_key_iv: "68579de6b20cd4462567cb2e474d329a"
server_key_version: "encryptionkey@0"
I20260619 18:41:39.268581 20109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "7748f42d7f1f4a198af8eda49775b80f"
format_stamp: "Formatted at 2026-06-19 18:41:39 on dist-test-slave-rxv6"
server_key: "b5cde50176cc625a9be4cecd7a9f081f"
server_key_iv: "68579de6b20cd4462567cb2e474d329a"
server_key_version: "encryptionkey@0"
I20260619 18:41:39.272080 20109 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 18:41:39.274410 20123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:39.275560 20109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:41:39.275669 20109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "7748f42d7f1f4a198af8eda49775b80f"
format_stamp: "Formatted at 2026-06-19 18:41:39 on dist-test-slave-rxv6"
server_key: "b5cde50176cc625a9be4cecd7a9f081f"
server_key_iv: "68579de6b20cd4462567cb2e474d329a"
server_key_version: "encryptionkey@0"
I20260619 18:41:39.275758 20109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:39.291715 20109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:39.292407 20109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:39.292586 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:39.299300 20109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44611
I20260619 18:41:39.299311 20175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44611 every 8 connection(s)
I20260619 18:41:39.300241 20109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:39.303299 20176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:39.308634 20176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f: Bootstrap starting.
I20260619 18:41:39.309175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 20109
I20260619 18:41:39.309279 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:39.309583 392 external_mini_cluster.cc:1428] Setting key 9fe7cf2b5ce64870b1cee4e750b52235
I20260619 18:41:39.311240 20176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:39.311933 20176 log.cc:824] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:39.313977 20176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f: No bootstrap required, opened a new log
I20260619 18:41:39.317981 20176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7748f42d7f1f4a198af8eda49775b80f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44611 } }
I20260619 18:41:39.318220 20176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:39.318269 20176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7748f42d7f1f4a198af8eda49775b80f, State: Initialized, Role: FOLLOWER
I20260619 18:41:39.318651 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.310847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38876 (local address 127.0.98.62:44611)
0619 18:41:39.311450 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:39.311464 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:39.311498 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:41:39.312125 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:39.312142 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:39.312615 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:39.312921 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:39.313245 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.314261 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.315413 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.315873 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.315985 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.316773 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:39.316778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:39.316791 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:39.317159 (+ 368us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:39.317213 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:39.317232 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:39.317303 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:39.317434 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 18:41:39.317718 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
I20260619 18:41:39.318670 20176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7748f42d7f1f4a198af8eda49775b80f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44611 } }
I20260619 18:41:39.318801 20176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:39.318859 20176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:39.318943 20176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:39.319890 20176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7748f42d7f1f4a198af8eda49775b80f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44611 } }
I20260619 18:41:39.320248 20176 leader_election.cc:302] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7748f42d7f1f4a198af8eda49775b80f; no voters:
I20260619 18:41:39.320564 20176 leader_election.cc:288] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:39.320650 20181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:39.320848 20181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [term 1 LEADER]: Becoming Leader. State: Replica: 7748f42d7f1f4a198af8eda49775b80f, State: Running, Role: LEADER
I20260619 18:41:39.321159 20181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7748f42d7f1f4a198af8eda49775b80f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44611 } }
I20260619 18:41:39.321730 20176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:39.322840 20183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7748f42d7f1f4a198af8eda49775b80f. Latest consensus state: current_term: 1 leader_uuid: "7748f42d7f1f4a198af8eda49775b80f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7748f42d7f1f4a198af8eda49775b80f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44611 } } }
I20260619 18:41:39.322868 20182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7748f42d7f1f4a198af8eda49775b80f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7748f42d7f1f4a198af8eda49775b80f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44611 } } }
I20260619 18:41:39.322976 20183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:39.322980 20182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:39.323364 20190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:39.326440 20190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:39.332351 20190 catalog_manager.cc:1347] Generated new cluster ID: 0d54e3aa78f14709a1fe116094ee464e
I20260619 18:41:39.332458 20190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:39.352188 20190 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:39.353164 20190 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:39.361129 20190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f: Generated new TSK 0
I20260619 18:41:39.361806 20190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:39.368405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44611
--builtin_ntp_servers=127.0.98.20:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 18:41:39.498476 20200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:39.498749 20200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:39.498787 20200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:39.502729 20200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:39.502782 20200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:39.502836 20200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 18:41:39.507550 20200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:39.508584 20200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:39.509882 20200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:39.519218 20206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:39.519197 20205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:39.519212 20208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:39.519464 20200 server_base.cc:1034] running on GCE node
I20260619 18:41:39.519907 20200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:39.520565 20200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:39.521732 20200 hybrid_clock.cc:648] HybridClock initialized: now 1781894499521709 us; error 37 us; skew 500 ppm
I20260619 18:41:39.523692 20200 webserver.cc:460] Webserver started at http://127.0.98.1:33867/ using document root <none> and password file <none>
I20260619 18:41:39.524273 20200 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:39.524353 20200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:39.524530 20200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:39.526634 20200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-0/data/instance:
uuid: "7541258c45f543c2881b53db7810452f"
format_stamp: "Formatted at 2026-06-19 18:41:39 on dist-test-slave-rxv6"
server_key: "d2600bf9e26171df6b039ca3ead36274"
server_key_iv: "ed278f240cd05bd74f3d793781edc019"
server_key_version: "encryptionkey@0"
I20260619 18:41:39.527145 20200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7541258c45f543c2881b53db7810452f"
format_stamp: "Formatted at 2026-06-19 18:41:39 on dist-test-slave-rxv6"
server_key: "d2600bf9e26171df6b039ca3ead36274"
server_key_iv: "ed278f240cd05bd74f3d793781edc019"
server_key_version: "encryptionkey@0"
I20260619 18:41:39.530848 20200 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:41:39.533342 20214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:39.534647 20200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:41:39.534752 20200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-0/wal
uuid: "7541258c45f543c2881b53db7810452f"
format_stamp: "Formatted at 2026-06-19 18:41:39 on dist-test-slave-rxv6"
server_key: "d2600bf9e26171df6b039ca3ead36274"
server_key_iv: "ed278f240cd05bd74f3d793781edc019"
server_key_version: "encryptionkey@0"
I20260619 18:41:39.534858 20200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:39.554867 20200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:39.555563 20200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:39.555734 20200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:39.556371 20200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:39.557556 20200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:39.557622 20200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:39.557675 20200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:39.557706 20200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:39.568058 20200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34263
I20260619 18:41:39.568078 20327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34263 every 8 connection(s)
I20260619 18:41:39.568991 20200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-0/data/info.pb
I20260619 18:41:39.575234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 20200
I20260619 18:41:39.575351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-0/wal/instance
I20260619 18:41:39.575650 392 external_mini_cluster.cc:1428] Setting key f84a21d3c84b5bf54129b689c0f9485e
I20260619 18:41:39.577680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44611
--builtin_ntp_servers=127.0.98.20:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:41:39.578269 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.570730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54859 (local address 127.0.98.62:44611)
0619 18:41:39.570845 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:39.570848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:39.571861 (+ 1013us) server_negotiation.cc:407] Connection header received
0619 18:41:39.573162 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:39.573164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:39.573213 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:39.573285 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:39.574211 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.574785 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.576172 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.576451 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.576567 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.577722 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:39.577726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:39.577729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:39.577742 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:39.577762 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:39.577770 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:39.577830 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:39.578065 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 18:41:39.578129 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:41:39.579099 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.571082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44611 (local address 127.0.98.1:54859)
0619 18:41:39.571713 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:39.571752 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:39.572967 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:39.573429 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:39.573436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:39.573517 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:39.574057 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:39.574072 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.574915 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.574918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:39.575952 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:39.575964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.576597 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.576601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:39.576788 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.577460 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:41:39.577462 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:39.577507 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:41:39.577578 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:39.577867 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:39.577876 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:39.577882 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:41:39.578019 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 18:41:39.578251 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":101,"threads_started":1}
I20260619 18:41:39.581260 20328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44611
I20260619 18:41:39.581508 20328 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:39.582067 20328 heartbeater.cc:511] Master 127.0.98.62:44611 requested a full tablet report, sending...
I20260619 18:41:39.583510 20140 ts_manager.cc:194] Registered new tserver with Master: 7541258c45f543c2881b53db7810452f (127.0.98.1:34263)
I20260619 18:41:39.584625 20140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:54859
W20260619 18:41:39.685631 20331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:39.685901 20331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:39.685978 20331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:39.690034 20331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:39.690097 20331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:39.690155 20331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 18:41:39.694957 20331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:39.695983 20331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:39.697321 20331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:39.706424 20336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:39.706444 20337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:39.706481 20331 server_base.cc:1034] running on GCE node
W20260619 18:41:39.706424 20339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:39.707079 20331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:39.707666 20331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:39.708873 20331 hybrid_clock.cc:648] HybridClock initialized: now 1781894499708853 us; error 32 us; skew 500 ppm
I20260619 18:41:39.710690 20331 webserver.cc:460] Webserver started at http://127.0.98.2:40651/ using document root <none> and password file <none>
I20260619 18:41:39.711226 20331 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:39.711292 20331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:39.711484 20331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:39.713507 20331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/data/instance:
uuid: "0c8c5ab1fec547b9ba7606de48a321eb"
format_stamp: "Formatted at 2026-06-19 18:41:39 on dist-test-slave-rxv6"
server_key: "7a0754f5c4d3a6ec38a5192fe43fdf5d"
server_key_iv: "774672e9ada228c5e5e719a82d649151"
server_key_version: "encryptionkey@0"
I20260619 18:41:39.714093 20331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0c8c5ab1fec547b9ba7606de48a321eb"
format_stamp: "Formatted at 2026-06-19 18:41:39 on dist-test-slave-rxv6"
server_key: "7a0754f5c4d3a6ec38a5192fe43fdf5d"
server_key_iv: "774672e9ada228c5e5e719a82d649151"
server_key_version: "encryptionkey@0"
I20260619 18:41:39.717748 20331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:41:39.720307 20345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:39.721681 20331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:41:39.721787 20331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/wal
uuid: "0c8c5ab1fec547b9ba7606de48a321eb"
format_stamp: "Formatted at 2026-06-19 18:41:39 on dist-test-slave-rxv6"
server_key: "7a0754f5c4d3a6ec38a5192fe43fdf5d"
server_key_iv: "774672e9ada228c5e5e719a82d649151"
server_key_version: "encryptionkey@0"
I20260619 18:41:39.721891 20331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:39.734680 20331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:39.735337 20331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:39.735502 20331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:39.736068 20331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:39.737018 20331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:39.737062 20331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:39.737104 20331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:39.737133 20331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:39.745149 20331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41137
I20260619 18:41:39.745213 20458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41137 every 8 connection(s)
I20260619 18:41:39.746080 20331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/data/info.pb
I20260619 18:41:39.754010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 20331
I20260619 18:41:39.754125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/wal/instance
I20260619 18:41:39.754411 392 external_mini_cluster.cc:1428] Setting key 502d7edfeef98cc6128f3305ce15f577
I20260619 18:41:39.754820 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.747808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40625 (local address 127.0.98.62:44611)
0619 18:41:39.747947 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:39.747951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:39.748943 (+ 992us) server_negotiation.cc:407] Connection header received
0619 18:41:39.750315 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:39.750319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:39.750378 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:39.750448 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:39.751322 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.751849 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.752964 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.753188 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.753278 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.754360 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:39.754364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:39.754366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:39.754378 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:39.754397 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:39.754405 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:39.754465 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:39.754616 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 18:41:39.754678 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 18:41:39.755765 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.748178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44611 (local address 127.0.98.2:40625)
0619 18:41:39.748799 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:39.748837 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:39.750103 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:39.750629 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:39.750636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:39.750720 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:39.751183 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:39.751195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.751968 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.751971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:39.752850 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:39.752858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.753326 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.753330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:39.753481 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.754087 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:41:39.754089 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:39.754142 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:41:39.754220 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:39.754474 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:39.754482 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:39.754488 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 18:41:39.754606 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 18:41:39.754875 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":100,"threads_started":1}
I20260619 18:41:39.756457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44611
--builtin_ntp_servers=127.0.98.20:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 18:41:39.756820 20459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44611
I20260619 18:41:39.757021 20459 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:39.757486 20459 heartbeater.cc:511] Master 127.0.98.62:44611 requested a full tablet report, sending...
I20260619 18:41:39.758556 20140 ts_manager.cc:194] Registered new tserver with Master: 0c8c5ab1fec547b9ba7606de48a321eb (127.0.98.2:41137)
I20260619 18:41:39.759104 20140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40625
W20260619 18:41:39.866842 20462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:39.867081 20462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:39.867117 20462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:39.871054 20462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:39.871105 20462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:39.871158 20462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 18:41:39.875869 20462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:39.876858 20462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:39.878214 20462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:39.887635 20468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:39.887717 20462 server_base.cc:1034] running on GCE node
W20260619 18:41:39.887642 20467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:39.887642 20470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:39.888267 20462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:39.888863 20462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:39.890009 20462 hybrid_clock.cc:648] HybridClock initialized: now 1781894499889986 us; error 33 us; skew 500 ppm
I20260619 18:41:39.892040 20462 webserver.cc:460] Webserver started at http://127.0.98.3:34471/ using document root <none> and password file <none>
I20260619 18:41:39.892592 20462 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:39.892659 20462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:39.892856 20462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:39.894888 20462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/data/instance:
uuid: "f1fae6948199455f9a6c55b737e65542"
format_stamp: "Formatted at 2026-06-19 18:41:39 on dist-test-slave-rxv6"
server_key: "b974bce400660502a0068409d7fd94cb"
server_key_iv: "387c98b1406c8abe2516478bee4c868d"
server_key_version: "encryptionkey@0"
I20260619 18:41:39.895404 20462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f1fae6948199455f9a6c55b737e65542"
format_stamp: "Formatted at 2026-06-19 18:41:39 on dist-test-slave-rxv6"
server_key: "b974bce400660502a0068409d7fd94cb"
server_key_iv: "387c98b1406c8abe2516478bee4c868d"
server_key_version: "encryptionkey@0"
I20260619 18:41:39.898931 20462 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:41:39.901147 20476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:39.902312 20462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:41:39.902412 20462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/wal
uuid: "f1fae6948199455f9a6c55b737e65542"
format_stamp: "Formatted at 2026-06-19 18:41:39 on dist-test-slave-rxv6"
server_key: "b974bce400660502a0068409d7fd94cb"
server_key_iv: "387c98b1406c8abe2516478bee4c868d"
server_key_version: "encryptionkey@0"
I20260619 18:41:39.902503 20462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:39.929739 20462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:39.930452 20462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:39.930619 20462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:39.931246 20462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 18:41:39.932559 20462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 18:41:39.932622 20462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:39.932686 20462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 18:41:39.932716 20462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:39.941629 20462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35471
I20260619 18:41:39.941687 20589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35471 every 8 connection(s)
I20260619 18:41:39.942548 20462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/data/info.pb
I20260619 18:41:39.951275 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.944304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49565 (local address 127.0.98.62:44611)
0619 18:41:39.944475 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:39.944480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:39.945508 (+ 1028us) server_negotiation.cc:407] Connection header received
0619 18:41:39.946869 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:39.946872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:39.946926 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:39.946998 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:39.947925 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.948454 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.949523 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.949749 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.949825 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.950871 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:39.950874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:39.950876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:39.950889 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:39.950907 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:39.950915 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:39.950975 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:39.951103 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 18:41:39.951154 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 18:41:39.952116 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.944730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44611 (local address 127.0.98.3:49565)
0619 18:41:39.945371 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:39.945411 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:39.946691 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:39.947193 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:39.947199 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:39.947285 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:41:39.947789 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:39.947802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.948549 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.948552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:39.949418 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:39.949428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.949867 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.949870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:39.950045 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.950622 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 18:41:39.950624 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:41:39.950663 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 18:41:39.950725 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:41:39.950982 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:41:39.950987 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:41:39.950992 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 18:41:39.951098 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 18:41:39.951319 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
I20260619 18:41:39.952821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 20462
I20260619 18:41:39.952939 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/wal/instance
I20260619 18:41:39.953229 392 external_mini_cluster.cc:1428] Setting key 935e96ce2a4c2f288a2cae23fdd7bee1
I20260619 18:41:39.953379 20590 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44611
I20260619 18:41:39.953625 20590 heartbeater.cc:464] Registering TS with master...
I20260619 18:41:39.954264 20590 heartbeater.cc:511] Master 127.0.98.62:44611 requested a full tablet report, sending...
I20260619 18:41:39.955274 20140 ts_manager.cc:194] Registered new tserver with Master: f1fae6948199455f9a6c55b737e65542 (127.0.98.3:35471)
I20260619 18:41:39.955819 20140 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:49565
I20260619 18:41:39.967216 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 18:41:39.972496 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.969110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38878 (local address 127.0.98.62:44611)
0619 18:41:39.969228 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:39.969231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:39.969378 (+ 147us) server_negotiation.cc:407] Connection header received
0619 18:41:39.969474 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:39.969476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:39.969517 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:39.969583 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:39.969812 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.970379 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.971232 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.971447 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.971521 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.972176 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:39.972179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:39.972181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:39.972192 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:39.972210 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:39.972217 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:39.972270 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:39.972328 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 18:41:39.972374 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:41:39.975062 20140 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 18:41:39.977983 20140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 18:41:39.991714 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.986686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58848 (local address 127.0.98.1:34263)
0619 18:41:39.987117 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:39.987129 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:39.987155 (+ 26us) server_negotiation.cc:407] Connection header received
0619 18:41:39.987237 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:39.987242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:39.987434 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:39.987562 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:39.987931 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.988797 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.990347 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.991068 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.991165 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.991357 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:39.991456 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 18:41:39.991546 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":109,"threads_started":1}
I20260619 18:41:39.991703 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.986512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34263 (local address 127.0.0.1:58848)
0619 18:41:39.986979 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:39.986991 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:39.987067 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:39.987583 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:39.987589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:39.987613 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:39.987817 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:39.987823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.988887 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.988891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:39.990222 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:39.990233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.991179 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.991182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:39.991283 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.991318 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 18:41:39.991426 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 18:41:39.991546 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":39,"thread_start_us":101,"threads_started":1}
I20260619 18:41:39.992062 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.985777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35471 (local address 127.0.0.1:60112)
0619 18:41:39.986598 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:39.986647 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:39.986767 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:39.988278 (+ 1511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:39.988281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:39.988300 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:39.988384 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:39.988389 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.989595 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.989598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:39.990695 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:39.990705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.991724 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.991727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:39.991816 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.991839 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 18:41:39.991884 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 18:41:39.991937 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":20,"thread_start_us":90,"threads_started":1}
I20260619 18:41:39.992197 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.985882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41137 (local address 127.0.0.1:49124)
0619 18:41:39.986874 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:39.986886 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:39.986957 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:39.987613 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:39.987617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:39.987644 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:39.987822 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:39.987830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.989222 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.989226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:39.990427 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:39.990433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.991501 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.991505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:39.991599 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.991623 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 18:41:39.991694 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 18:41:39.992081 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933,"mutex_wait_us":79,"spinlock_wait_cycles":3840,"thread_start_us":106,"threads_started":1}
I20260619 18:41:39.992595 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.986099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49124 (local address 127.0.98.2:41137)
0619 18:41:39.987199 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:39.987211 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:39.987235 (+ 24us) server_negotiation.cc:407] Connection header received
0619 18:41:39.987310 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:39.987315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:39.987471 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:39.987606 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:39.987956 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.989111 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.990523 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.991177 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.991275 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.991439 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:39.992333 (+ 894us) server_negotiation.cc:299] Negotiation successful
0619 18:41:39.992428 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":996,"thread_start_us":102,"threads_started":1}
I20260619 18:41:39.993671 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:39.986334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60112 (local address 127.0.98.3:35471)
0619 18:41:39.987852 (+ 1518us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:39.987863 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:39.987885 (+ 22us) server_negotiation.cc:407] Connection header received
0619 18:41:39.987955 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:39.987960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:39.988144 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:39.988285 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:39.988475 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.989475 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.990815 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:39.991599 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:39.992426 (+ 827us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:39.992605 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:39.993425 (+ 820us) server_negotiation.cc:299] Negotiation successful
0619 18:41:39.993515 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1417,"thread_start_us":152,"threads_started":1}
I20260619 18:41:39.994817 20262 tablet_service.cc:1467] Processing CreateTablet for tablet eb8da0756f94423182f16ee88ab8ab06 (DEFAULT_TABLE table=test-table [id=6e90e8d7e0944511a19937b38cca380e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:39.995028 20393 tablet_service.cc:1467] Processing CreateTablet for tablet eb8da0756f94423182f16ee88ab8ab06 (DEFAULT_TABLE table=test-table [id=6e90e8d7e0944511a19937b38cca380e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:39.995766 20262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb8da0756f94423182f16ee88ab8ab06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:39.996047 20393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb8da0756f94423182f16ee88ab8ab06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:39.996155 20524 tablet_service.cc:1467] Processing CreateTablet for tablet eb8da0756f94423182f16ee88ab8ab06 (DEFAULT_TABLE table=test-table [id=6e90e8d7e0944511a19937b38cca380e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 18:41:39.997121 20524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb8da0756f94423182f16ee88ab8ab06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:40.001420 20607 tablet_bootstrap.cc:492] T eb8da0756f94423182f16ee88ab8ab06 P 7541258c45f543c2881b53db7810452f: Bootstrap starting.
I20260619 18:41:40.003298 20607 tablet_bootstrap.cc:654] T eb8da0756f94423182f16ee88ab8ab06 P 7541258c45f543c2881b53db7810452f: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:40.003669 20609 tablet_bootstrap.cc:492] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542: Bootstrap starting.
I20260619 18:41:40.004279 20607 log.cc:824] T eb8da0756f94423182f16ee88ab8ab06 P 7541258c45f543c2881b53db7810452f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:40.004971 20608 tablet_bootstrap.cc:492] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb: Bootstrap starting.
I20260619 18:41:40.005713 20609 tablet_bootstrap.cc:654] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:40.006266 20607 tablet_bootstrap.cc:492] T eb8da0756f94423182f16ee88ab8ab06 P 7541258c45f543c2881b53db7810452f: No bootstrap required, opened a new log
I20260619 18:41:40.006462 20607 ts_tablet_manager.cc:1397] T eb8da0756f94423182f16ee88ab8ab06 P 7541258c45f543c2881b53db7810452f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 18:41:40.006578 20609 log.cc:824] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:40.006973 20608 tablet_bootstrap.cc:654] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:40.007659 20608 log.cc:824] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:40.008553 20609 tablet_bootstrap.cc:492] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542: No bootstrap required, opened a new log
I20260619 18:41:40.008742 20609 ts_tablet_manager.cc:1397] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:41:40.009196 20608 tablet_bootstrap.cc:492] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb: No bootstrap required, opened a new log
I20260619 18:41:40.009361 20608 ts_tablet_manager.cc:1397] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 18:41:40.010978 20607 raft_consensus.cc:348] T eb8da0756f94423182f16ee88ab8ab06 P 7541258c45f543c2881b53db7810452f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1fae6948199455f9a6c55b737e65542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35471 } } peers { permanent_uuid: "0c8c5ab1fec547b9ba7606de48a321eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41137 } } peers { permanent_uuid: "7541258c45f543c2881b53db7810452f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } }
I20260619 18:41:40.011276 20607 raft_consensus.cc:374] T eb8da0756f94423182f16ee88ab8ab06 P 7541258c45f543c2881b53db7810452f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:40.011334 20607 raft_consensus.cc:729] T eb8da0756f94423182f16ee88ab8ab06 P 7541258c45f543c2881b53db7810452f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7541258c45f543c2881b53db7810452f, State: Initialized, Role: FOLLOWER
I20260619 18:41:40.011768 20607 consensus_queue.cc:260] T eb8da0756f94423182f16ee88ab8ab06 P 7541258c45f543c2881b53db7810452f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1fae6948199455f9a6c55b737e65542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35471 } } peers { permanent_uuid: "0c8c5ab1fec547b9ba7606de48a321eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41137 } } peers { permanent_uuid: "7541258c45f543c2881b53db7810452f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } }
I20260619 18:41:40.012403 20328 heartbeater.cc:503] Master 127.0.98.62:44611 was elected leader, sending a full tablet report...
I20260619 18:41:40.012776 20607 ts_tablet_manager.cc:1428] T eb8da0756f94423182f16ee88ab8ab06 P 7541258c45f543c2881b53db7810452f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 18:41:40.012704 20609 raft_consensus.cc:348] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1fae6948199455f9a6c55b737e65542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35471 } } peers { permanent_uuid: "0c8c5ab1fec547b9ba7606de48a321eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41137 } } peers { permanent_uuid: "7541258c45f543c2881b53db7810452f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } }
I20260619 18:41:40.012946 20609 raft_consensus.cc:374] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:40.013003 20609 raft_consensus.cc:729] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1fae6948199455f9a6c55b737e65542, State: Initialized, Role: FOLLOWER
I20260619 18:41:40.013386 20609 consensus_queue.cc:260] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1fae6948199455f9a6c55b737e65542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35471 } } peers { permanent_uuid: "0c8c5ab1fec547b9ba7606de48a321eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41137 } } peers { permanent_uuid: "7541258c45f543c2881b53db7810452f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } }
I20260619 18:41:40.013490 20608 raft_consensus.cc:348] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1fae6948199455f9a6c55b737e65542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35471 } } peers { permanent_uuid: "0c8c5ab1fec547b9ba7606de48a321eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41137 } } peers { permanent_uuid: "7541258c45f543c2881b53db7810452f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } }
I20260619 18:41:40.013721 20608 raft_consensus.cc:374] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:40.013777 20608 raft_consensus.cc:729] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8c5ab1fec547b9ba7606de48a321eb, State: Initialized, Role: FOLLOWER
I20260619 18:41:40.014137 20590 heartbeater.cc:503] Master 127.0.98.62:44611 was elected leader, sending a full tablet report...
I20260619 18:41:40.014324 20609 ts_tablet_manager.cc:1428] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 18:41:40.015057 20608 consensus_queue.cc:260] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1fae6948199455f9a6c55b737e65542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35471 } } peers { permanent_uuid: "0c8c5ab1fec547b9ba7606de48a321eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41137 } } peers { permanent_uuid: "7541258c45f543c2881b53db7810452f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } }
I20260619 18:41:40.015674 20459 heartbeater.cc:503] Master 127.0.98.62:44611 was elected leader, sending a full tablet report...
I20260619 18:41:40.016994 20608 ts_tablet_manager.cc:1428] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260619 18:41:40.070108 20329 tablet.cc:2369] T eb8da0756f94423182f16ee88ab8ab06 P 7541258c45f543c2881b53db7810452f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:41:40.130199 20614 raft_consensus.cc:482] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 18:41:40.130409 20614 raft_consensus.cc:504] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1fae6948199455f9a6c55b737e65542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35471 } } peers { permanent_uuid: "0c8c5ab1fec547b9ba7606de48a321eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41137 } } peers { permanent_uuid: "7541258c45f543c2881b53db7810452f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } }
I20260619 18:41:40.131564 20614 leader_election.cc:288] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c8c5ab1fec547b9ba7606de48a321eb (127.0.98.2:41137), 7541258c45f543c2881b53db7810452f (127.0.98.1:34263)
I20260619 18:41:40.135052 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:40.131928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46839 (local address 127.0.98.1:34263)
0619 18:41:40.132042 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:40.132046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:40.132060 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:41:40.132180 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:40.132184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:40.132228 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:40.132292 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:40.132580 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:40.133088 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:40.134137 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:40.134550 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:40.134655 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:40.134717 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:40.134868 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 18:41:40.134937 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:41:40.135052 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:40.131836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34263 (local address 127.0.98.3:46839)
0619 18:41:40.131954 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:40.131970 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:40.132066 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:40.132314 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:40.132317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:40.132334 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:40.132456 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:40.132462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:40.133200 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:40.133204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:40.134018 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:40.134027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:40.134664 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:40.134668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:40.134763 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:40.134783 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 18:41:40.134845 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 18:41:40.134915 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"spinlock_wait_cycles":19840}
I20260619 18:41:40.135402 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:40.131916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37153 (local address 127.0.98.2:41137)
0619 18:41:40.132029 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:40.132034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:40.132356 (+ 322us) server_negotiation.cc:407] Connection header received
0619 18:41:40.132482 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:40.132485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:40.132541 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:40.132604 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 18:41:40.132869 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:40.133376 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:40.134470 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:40.134868 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:40.134958 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:40.135012 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:40.135218 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 18:41:40.135290 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:41:40.135723 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:40.131837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41137 (local address 127.0.98.3:37153)
0619 18:41:40.132260 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:40.132271 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:41:40.132381 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:41:40.132623 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:41:40.132626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:41:40.132640 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 18:41:40.132755 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:40.132760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:40.133500 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:40.133503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:40.134352 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:41:40.134361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:41:40.134983 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:41:40.134987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:41:40.135073 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:40.135089 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 18:41:40.135539 (+ 450us) client_negotiation.cc:240] Negotiation successful
0619 18:41:40.135610 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":110,"thread_start_us":108,"threads_started":1}
I20260619 18:41:40.135690 20282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8da0756f94423182f16ee88ab8ab06" candidate_uuid: "f1fae6948199455f9a6c55b737e65542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7541258c45f543c2881b53db7810452f" is_pre_election: true
I20260619 18:41:40.135984 20282 raft_consensus.cc:2454] T eb8da0756f94423182f16ee88ab8ab06 P 7541258c45f543c2881b53db7810452f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1fae6948199455f9a6c55b737e65542 in term 0.
I20260619 18:41:40.136274 20413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8da0756f94423182f16ee88ab8ab06" candidate_uuid: "f1fae6948199455f9a6c55b737e65542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8c5ab1fec547b9ba7606de48a321eb" is_pre_election: true
I20260619 18:41:40.136471 20478 leader_election.cc:302] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7541258c45f543c2881b53db7810452f, f1fae6948199455f9a6c55b737e65542; no voters:
I20260619 18:41:40.136509 20413 raft_consensus.cc:2454] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1fae6948199455f9a6c55b737e65542 in term 0.
I20260619 18:41:40.136722 20614 raft_consensus.cc:2781] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 18:41:40.136813 20614 raft_consensus.cc:482] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 18:41:40.136864 20614 raft_consensus.cc:3037] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:40.137811 20614 raft_consensus.cc:504] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1fae6948199455f9a6c55b737e65542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35471 } } peers { permanent_uuid: "0c8c5ab1fec547b9ba7606de48a321eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41137 } } peers { permanent_uuid: "7541258c45f543c2881b53db7810452f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } }
I20260619 18:41:40.138147 20614 leader_election.cc:288] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [CANDIDATE]: Term 1 election: Requested vote from peers 0c8c5ab1fec547b9ba7606de48a321eb (127.0.98.2:41137), 7541258c45f543c2881b53db7810452f (127.0.98.1:34263)
I20260619 18:41:40.138443 20413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8da0756f94423182f16ee88ab8ab06" candidate_uuid: "f1fae6948199455f9a6c55b737e65542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c8c5ab1fec547b9ba7606de48a321eb"
I20260619 18:41:40.138491 20282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb8da0756f94423182f16ee88ab8ab06" candidate_uuid: "f1fae6948199455f9a6c55b737e65542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7541258c45f543c2881b53db7810452f"
I20260619 18:41:40.138576 20413 raft_consensus.cc:3037] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:40.138604 20282 raft_consensus.cc:3037] T eb8da0756f94423182f16ee88ab8ab06 P 7541258c45f543c2881b53db7810452f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:40.139556 20413 raft_consensus.cc:2454] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1fae6948199455f9a6c55b737e65542 in term 1.
I20260619 18:41:40.139592 20282 raft_consensus.cc:2454] T eb8da0756f94423182f16ee88ab8ab06 P 7541258c45f543c2881b53db7810452f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1fae6948199455f9a6c55b737e65542 in term 1.
I20260619 18:41:40.139849 20480 leader_election.cc:302] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8c5ab1fec547b9ba7606de48a321eb, f1fae6948199455f9a6c55b737e65542; no voters:
I20260619 18:41:40.140035 20614 raft_consensus.cc:2781] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:40.140273 20614 raft_consensus.cc:686] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [term 1 LEADER]: Becoming Leader. State: Replica: f1fae6948199455f9a6c55b737e65542, State: Running, Role: LEADER
I20260619 18:41:40.140590 20614 consensus_queue.cc:237] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1fae6948199455f9a6c55b737e65542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35471 } } peers { permanent_uuid: "0c8c5ab1fec547b9ba7606de48a321eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41137 } } peers { permanent_uuid: "7541258c45f543c2881b53db7810452f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } }
I20260619 18:41:40.143618 20140 catalog_manager.cc:5495] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1fae6948199455f9a6c55b737e65542 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f1fae6948199455f9a6c55b737e65542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1fae6948199455f9a6c55b737e65542" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c8c5ab1fec547b9ba7606de48a321eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7541258c45f543c2881b53db7810452f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 } health_report { overall_health: UNKNOWN } } }
I20260619 18:41:40.186748 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:40.183420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60120 (local address 127.0.98.3:35471)
0619 18:41:40.183551 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:40.183555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:40.183569 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:41:40.183693 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:40.183696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:40.183749 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:40.183837 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:41:40.184054 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:40.184544 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:40.185358 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:40.185574 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:40.185657 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:40.185968 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:41:40.186382 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:41:40.186487 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:40.186564 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 18:41:40.186614 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 18:41:40.192287 20282 raft_consensus.cc:1261] T eb8da0756f94423182f16ee88ab8ab06 P 7541258c45f543c2881b53db7810452f [term 1 FOLLOWER]: Refusing update from remote peer f1fae6948199455f9a6c55b737e65542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:40.192287 20413 raft_consensus.cc:1261] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb [term 1 FOLLOWER]: Refusing update from remote peer f1fae6948199455f9a6c55b737e65542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:41:40.192973 20614 consensus_queue.cc:1035] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c8c5ab1fec547b9ba7606de48a321eb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:41:40.193161 20617 consensus_queue.cc:1035] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7541258c45f543c2881b53db7810452f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 18:41:40.193439 20591 tablet.cc:2369] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:41:40.199188 20620 mvcc.cc:205] Tried to move back new op lower bound from 7298639872780767232 to 7298639872578605056. Current Snapshot: MvccSnapshot[applied={T|T < 7298639872780767232}]
I20260619 18:41:40.209748 20140 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:40.209921 20140 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 18:41:40.212344 20140 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7748f42d7f1f4a198af8eda49775b80f: Sending DeleteTablet for 3 replicas of tablet eb8da0756f94423182f16ee88ab8ab06
I20260619 18:41:40.213107 20524 tablet_service.cc:1514] Processing DeleteTablet for tablet eb8da0756f94423182f16ee88ab8ab06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:40 UTC) from {username='slave'} at 127.0.0.1:60112
I20260619 18:41:40.213107 20393 tablet_service.cc:1514] Processing DeleteTablet for tablet eb8da0756f94423182f16ee88ab8ab06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:40 UTC) from {username='slave'} at 127.0.0.1:49124
I20260619 18:41:40.213230 20262 tablet_service.cc:1514] Processing DeleteTablet for tablet eb8da0756f94423182f16ee88ab8ab06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 18:41:40 UTC) from {username='slave'} at 127.0.0.1:58848
I20260619 18:41:40.213624 20630 tablet_replica.cc:330] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542: stopping tablet replica
I20260619 18:41:40.213626 20631 tablet_replica.cc:330] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb: stopping tablet replica
I20260619 18:41:40.213960 20630 raft_consensus.cc:2229] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [term 1 LEADER]: Raft consensus shutting down.
I20260619 18:41:40.214120 20631 raft_consensus.cc:2229] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 18:41:40.214323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 20200
I20260619 18:41:40.214385 20630 raft_consensus.cc:2258] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:40.214562 20631 raft_consensus.cc:2258] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 18:41:40.216132 20630 ts_tablet_manager.cc:1905] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:40.218971 20630 ts_tablet_manager.cc:1918] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:40.219117 20630 log.cc:1196] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-2/wal/wals/eb8da0756f94423182f16ee88ab8ab06
I20260619 18:41:40.219158 20631 ts_tablet_manager.cc:1905] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 18:41:40.219545 20630 ts_tablet_manager.cc:1939] T eb8da0756f94423182f16ee88ab8ab06 P f1fae6948199455f9a6c55b737e65542: Deleting consensus metadata
I20260619 18:41:40.220863 20125 catalog_manager.cc:4841] TS f1fae6948199455f9a6c55b737e65542 (127.0.98.3:35471): tablet eb8da0756f94423182f16ee88ab8ab06 (table test-table [id=6e90e8d7e0944511a19937b38cca380e]) successfully deleted
W20260619 18:41:40.221611 20125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34263 (error 108)
I20260619 18:41:40.221824 20631 ts_tablet_manager.cc:1918] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 18:41:40.221944 20631 log.cc:1196] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb: Deleting WAL directory at /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781894249862417-392-0/minicluster-data/ts-1/wal/wals/eb8da0756f94423182f16ee88ab8ab06
I20260619 18:41:40.222268 20631 ts_tablet_manager.cc:1939] T eb8da0756f94423182f16ee88ab8ab06 P 0c8c5ab1fec547b9ba7606de48a321eb: Deleting consensus metadata
I20260619 18:41:40.222618 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 20331
I20260619 18:41:40.223620 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:40.222922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34263 (local address 127.0.0.1:58856)
0619 18:41:40.223450 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:40.223532 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438}
W20260619 18:41:40.226048 20125 catalog_manager.cc:4584] TS 7541258c45f543c2881b53db7810452f (127.0.98.1:34263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eb8da0756f94423182f16ee88ab8ab06: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34263: connect: Connection refused (error 111)
I20260619 18:41:40.227807 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 20462
I20260619 18:41:40.228289 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:40.228020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41137 (local address 127.0.0.1:49132)
0619 18:41:40.228168 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 18:41:40.228212 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 18:41:40.232978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 20109
2026-06-19T18:41:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1126 ms)
[----------] 6 tests from EncryptionPolicyTest (10245 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:40 dist-test-slave-rxv6 krb5kdc[20636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:40 dist-test-slave-rxv6 krb5kdc[20636](info): set up 2 sockets
Jun 19 18:41:40 dist-test-slave-rxv6 krb5kdc[20636](info): commencing operation
krb5kdc: starting...
W20260619 18:41:41.185858 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:41 dist-test-slave-rxv6 krb5kdc[20636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:41:41.303054 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38111
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:41.414238 20649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:41.414515 20649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:41.414577 20649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:41.418565 20649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:41.418627 20649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:41.418653 20649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:41.418674 20649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:41.418699 20649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:41.424028 20649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38111
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:41.425060 20649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:41.426069 20649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:41.434432 20655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:41.434432 20654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:41.434492 20649 server_base.cc:1034] running on GCE node
W20260619 18:41:41.434453 20657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:41.435286 20649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:41.436547 20649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:41.437723 20649 hybrid_clock.cc:648] HybridClock initialized: now 1781894501437693 us; error 41 us; skew 500 ppm
Jun 19 18:41:41 dist-test-slave-rxv6 krb5kdc[20636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894501, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:41.443753 20649 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:41:41.444720 20649 webserver.cc:460] Webserver started at http://127.0.0.1:40317/ using document root <none> and password file <none>
I20260619 18:41:41.445461 20649 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:41.445556 20649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:41.445832 20649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:41.448082 20649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "324171868a144b0c8f5de23da84f4511"
format_stamp: "Formatted at 2026-06-19 18:41:41 on dist-test-slave-rxv6"
server_key: "6495f55a6f9ddcb5722dfabec76270c4"
server_key_iv: "1e8586c5683ea6cf9cadc01a0c71b23d"
server_key_version: "encryptionkey@0"
I20260619 18:41:41.448670 20649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "324171868a144b0c8f5de23da84f4511"
format_stamp: "Formatted at 2026-06-19 18:41:41 on dist-test-slave-rxv6"
server_key: "6495f55a6f9ddcb5722dfabec76270c4"
server_key_iv: "1e8586c5683ea6cf9cadc01a0c71b23d"
server_key_version: "encryptionkey@0"
I20260619 18:41:41.452845 20649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:41:41.455587 20664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:41.456945 20649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:41:41.457072 20649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "324171868a144b0c8f5de23da84f4511"
format_stamp: "Formatted at 2026-06-19 18:41:41 on dist-test-slave-rxv6"
server_key: "6495f55a6f9ddcb5722dfabec76270c4"
server_key_iv: "1e8586c5683ea6cf9cadc01a0c71b23d"
server_key_version: "encryptionkey@0"
I20260619 18:41:41.457178 20649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:41.467622 20649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:41.470400 20649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:41.470571 20649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:41.477085 20649 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38111
I20260619 18:41:41.477100 20716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38111 every 8 connection(s)
I20260619 18:41:41.478047 20649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:41.479424 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 20649
I20260619 18:41:41.479553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:41.479864 392 external_mini_cluster.cc:1428] Setting key 4ebfdf7045b7f69f5807d094ed485aee
I20260619 18:41:41.481187 20717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:41.487355 20717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511: Bootstrap starting.
Jun 19 18:41:41 dist-test-slave-rxv6 krb5kdc[20636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:41:41.489756 20717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:41.490566 20717 log.cc:824] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:41.492358 20717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511: No bootstrap required, opened a new log
I20260619 18:41:41.493836 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:41.481155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37629 (local address 127.0.0.1:38111)
0619 18:41:41.481563 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:41.481577 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:41.481612 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:41:41.482328 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:41.482345 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:41.482672 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:41.482950 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:41.483836 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:41.484802 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:41.485988 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:41.486444 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:41.486557 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:41.489164 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:41.489169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:41.489181 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:41.489207 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:41.490917 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:41.491376 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:41.491380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:41.491385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:41.491488 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:41.491776 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:41.491779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:41.491780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:41.492156 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:41.492352 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:41.492661 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 18:41:41.492908 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":111,"threads_started":1}
I20260619 18:41:41.496562 20717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324171868a144b0c8f5de23da84f4511" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } }
I20260619 18:41:41.496841 20717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:41.496925 20717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324171868a144b0c8f5de23da84f4511, State: Initialized, Role: FOLLOWER
I20260619 18:41:41.496930 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:41:41.497344 20717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324171868a144b0c8f5de23da84f4511" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } }
I20260619 18:41:41.497466 20717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:41.497525 20717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:41.497627 20717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:41.498669 20717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324171868a144b0c8f5de23da84f4511" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } }
I20260619 18:41:41.499060 20717 leader_election.cc:302] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 324171868a144b0c8f5de23da84f4511; no voters:
I20260619 18:41:41.499439 20717 leader_election.cc:288] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:41.499522 20727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:41.499704 20727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [term 1 LEADER]: Becoming Leader. State: Replica: 324171868a144b0c8f5de23da84f4511, State: Running, Role: LEADER
I20260619 18:41:41.500069 20727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324171868a144b0c8f5de23da84f4511" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } }
I20260619 18:41:41.500821 20717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:41.501688 20728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "324171868a144b0c8f5de23da84f4511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324171868a144b0c8f5de23da84f4511" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } } }
I20260619 18:41:41.502036 20728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:41.502331 20728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 324171868a144b0c8f5de23da84f4511. Latest consensus state: current_term: 1 leader_uuid: "324171868a144b0c8f5de23da84f4511" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324171868a144b0c8f5de23da84f4511" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38111 } } }
I20260619 18:41:41.502483 20728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:41.503127 20736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:41.506083 20736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:41.507697 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:41.498960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53539 (local address 127.0.0.1:38111)
0619 18:41:41.499072 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:41.499077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:41.499300 (+ 223us) server_negotiation.cc:407] Connection header received
0619 18:41:41.499455 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:41.499458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:41.499506 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:41.499576 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:41.500444 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:41.501076 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:41.502191 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:41.502523 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:41.502597 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:41.504139 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:41.504142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:41.504144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:41.504167 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:41.505712 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:41.506298 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:41.506302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:41.506304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:41.506391 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:41.506762 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:41.506766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:41.506767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:41.507001 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:41.507139 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:41.507441 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 18:41:41.507543 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 18:41:41.511987 20736 catalog_manager.cc:1347] Generated new cluster ID: 39d4effbe0fd46a8a7f0cfcf87d234cd
I20260619 18:41:41.512069 20736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:41.525565 20736 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:41.526707 20736 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:41.532828 20736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 324171868a144b0c8f5de23da84f4511: Generated new TSK 0
I20260619 18:41:41.533582 20736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:41.539477 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 20649
W20260619 18:41:41.544031 20634 connection.cc:535] client connection to 127.0.0.1:38111 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38111): Connection reset by peer (error 104)
2026-06-19T18:41:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:41 dist-test-slave-rxv6 krb5kdc[20748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:41 dist-test-slave-rxv6 krb5kdc[20748](info): set up 2 sockets
Jun 19 18:41:41 dist-test-slave-rxv6 krb5kdc[20748](info): commencing operation
krb5kdc: starting...
W20260619 18:41:42.489789 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:42 dist-test-slave-rxv6 krb5kdc[20748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:41:42.609215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41667
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:42.718688 20761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:42.718986 20761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:42.719046 20761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:42.723063 20761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:42.723138 20761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:42.723166 20761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:42.723186 20761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:42.723213 20761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:42.728579 20761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41667
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:42.729669 20761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:42.730734 20761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:42.739208 20766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:42.739244 20769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:42.739408 20761 server_base.cc:1034] running on GCE node
W20260619 18:41:42.739203 20767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:42.740063 20761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:42.741212 20761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:42.742375 20761 hybrid_clock.cc:648] HybridClock initialized: now 1781894502742349 us; error 39 us; skew 500 ppm
Jun 19 18:41:42 dist-test-slave-rxv6 krb5kdc[20748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:42.745206 20761 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:41:42.746228 20761 webserver.cc:460] Webserver started at http://127.0.0.1:33371/ using document root <none> and password file <none>
I20260619 18:41:42.746773 20761 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:42.746842 20761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:42.747044 20761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:42.749111 20761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "4c7d4bc20b7d450988c2ccb9f55b84e8"
format_stamp: "Formatted at 2026-06-19 18:41:42 on dist-test-slave-rxv6"
server_key: "2f23048bfa567218170eefc68f087e0a"
server_key_iv: "78a6eecae7a2811ef4e8e163d69222e7"
server_key_version: "encryptionkey@0"
I20260619 18:41:42.749642 20761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c7d4bc20b7d450988c2ccb9f55b84e8"
format_stamp: "Formatted at 2026-06-19 18:41:42 on dist-test-slave-rxv6"
server_key: "2f23048bfa567218170eefc68f087e0a"
server_key_iv: "78a6eecae7a2811ef4e8e163d69222e7"
server_key_version: "encryptionkey@0"
I20260619 18:41:42.753409 20761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 18:41:42.756043 20776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:42.757296 20761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 18:41:42.757405 20761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "4c7d4bc20b7d450988c2ccb9f55b84e8"
format_stamp: "Formatted at 2026-06-19 18:41:42 on dist-test-slave-rxv6"
server_key: "2f23048bfa567218170eefc68f087e0a"
server_key_iv: "78a6eecae7a2811ef4e8e163d69222e7"
server_key_version: "encryptionkey@0"
I20260619 18:41:42.757515 20761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:42.773238 20761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:42.776288 20761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:42.776477 20761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:42.783126 20761 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41667
I20260619 18:41:42.783174 20828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41667 every 8 connection(s)
I20260619 18:41:42.784067 20761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:42.785745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 20761
I20260619 18:41:42.785878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:42.786247 392 external_mini_cluster.cc:1428] Setting key 05092ea1d07c58323d24c5eca5225420
I20260619 18:41:42.787133 20829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:42.793265 20829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8: Bootstrap starting.
I20260619 18:41:42.794909 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:42.787607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33591 (local address 127.0.0.1:41667)
0619 18:41:42.787994 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:42.788016 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:42.788049 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:41:42.788666 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:42.788682 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:42.789020 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:42.789291 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:42.789618 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:42.790616 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:42.791752 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:42.792227 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:42.792336 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:42.793227 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:42.793232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:42.793245 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:42.793648 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:42.793699 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:42.793722 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:42.793818 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:42.794050 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 18:41:42.794180 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":111,"threads_started":1}
I20260619 18:41:42.795679 20829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:42.796406 20829 log.cc:824] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:42.797807 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:41:42.798408 20829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8: No bootstrap required, opened a new log
I20260619 18:41:42.802023 20829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7d4bc20b7d450988c2ccb9f55b84e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41667 } }
I20260619 18:41:42.802299 20829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:42.802359 20829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c7d4bc20b7d450988c2ccb9f55b84e8, State: Initialized, Role: FOLLOWER
I20260619 18:41:42.802848 20829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7d4bc20b7d450988c2ccb9f55b84e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41667 } }
I20260619 18:41:42.802994 20829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:42.803099 20829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:42.803212 20829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:42.803716 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:42.799706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32939 (local address 127.0.0.1:41667)
0619 18:41:42.799810 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:42.799814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:42.799996 (+ 182us) server_negotiation.cc:407] Connection header received
0619 18:41:42.800129 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:42.800131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:42.800167 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:42.800221 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:42.800589 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:42.801060 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:42.802159 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:42.802398 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:42.802491 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:42.803277 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:42.803280 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:42.803282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:42.803294 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:42.803323 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:42.803336 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:42.803411 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:42.803558 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 18:41:42.803590 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 18:41:42.804302 20829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7d4bc20b7d450988c2ccb9f55b84e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41667 } }
I20260619 18:41:42.804657 20829 leader_election.cc:302] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c7d4bc20b7d450988c2ccb9f55b84e8; no voters:
I20260619 18:41:42.804977 20829 leader_election.cc:288] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:42.805073 20839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:42.805267 20839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [term 1 LEADER]: Becoming Leader. State: Replica: 4c7d4bc20b7d450988c2ccb9f55b84e8, State: Running, Role: LEADER
I20260619 18:41:42.805644 20839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7d4bc20b7d450988c2ccb9f55b84e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41667 } }
I20260619 18:41:42.806187 20829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:42.807260 20840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c7d4bc20b7d450988c2ccb9f55b84e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7d4bc20b7d450988c2ccb9f55b84e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41667 } } }
I20260619 18:41:42.807269 20841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c7d4bc20b7d450988c2ccb9f55b84e8. Latest consensus state: current_term: 1 leader_uuid: "4c7d4bc20b7d450988c2ccb9f55b84e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7d4bc20b7d450988c2ccb9f55b84e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41667 } } }
I20260619 18:41:42.807430 20841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:42.807657 20840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:42.807852 20848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:42.811058 20848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:42.816334 20848 catalog_manager.cc:1347] Generated new cluster ID: e0380955c22948ae9f7bf3bd2c96e09e
I20260619 18:41:42.816421 20848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:42.832736 20848 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:42.834110 20848 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:42.838624 20848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c7d4bc20b7d450988c2ccb9f55b84e8: Generated new TSK 0
I20260619 18:41:42.839283 20848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:42.845717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 20761
W20260619 18:41:42.850121 20746 connection.cc:535] client connection to 127.0.0.1:41667 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41667): Connection reset by peer (error 104)
2026-06-19T18:41:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:42 dist-test-slave-rxv6 krb5kdc[20860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:42 dist-test-slave-rxv6 krb5kdc[20860](info): set up 2 sockets
Jun 19 18:41:42 dist-test-slave-rxv6 krb5kdc[20860](info): commencing operation
krb5kdc: starting...
W20260619 18:41:43.809804 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:43 dist-test-slave-rxv6 krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:41:43.925278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:44.034179 20873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:44.034461 20873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:44.034526 20873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:44.038614 20873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:44.038686 20873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:44.038712 20873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:44.038733 20873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:44.038759 20873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:44.044164 20873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43983
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:44.045266 20873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:44.046314 20873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:44.054391 20881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:44.054593 20879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:44.054417 20873 server_base.cc:1034] running on GCE node
W20260619 18:41:44.054416 20878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:44.055239 20873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:44.056314 20873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:44.057480 20873 hybrid_clock.cc:648] HybridClock initialized: now 1781894504057467 us; error 28 us; skew 500 ppm
Jun 19 18:41:44 dist-test-slave-rxv6 krb5kdc[20860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:44.060211 20873 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:41:44.061074 20873 webserver.cc:460] Webserver started at http://127.0.0.1:42867/ using document root <none> and password file <none>
I20260619 18:41:44.061585 20873 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:44.061640 20873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:44.061800 20873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:44.063817 20873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "e3532e301e2f4d90810aa1c2b2236c76"
format_stamp: "Formatted at 2026-06-19 18:41:44 on dist-test-slave-rxv6"
server_key: "7351d7578b7d170609d73f9cdb08a25d"
server_key_iv: "4bf8c77c96a3cc7ba6bff920c54240e4"
server_key_version: "encryptionkey@0"
I20260619 18:41:44.064319 20873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "e3532e301e2f4d90810aa1c2b2236c76"
format_stamp: "Formatted at 2026-06-19 18:41:44 on dist-test-slave-rxv6"
server_key: "7351d7578b7d170609d73f9cdb08a25d"
server_key_iv: "4bf8c77c96a3cc7ba6bff920c54240e4"
server_key_version: "encryptionkey@0"
I20260619 18:41:44.067924 20873 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:41:44.070412 20888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:44.071667 20873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:41:44.071792 20873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "e3532e301e2f4d90810aa1c2b2236c76"
format_stamp: "Formatted at 2026-06-19 18:41:44 on dist-test-slave-rxv6"
server_key: "7351d7578b7d170609d73f9cdb08a25d"
server_key_iv: "4bf8c77c96a3cc7ba6bff920c54240e4"
server_key_version: "encryptionkey@0"
I20260619 18:41:44.071899 20873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:44.076498 20873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:44.079613 20873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:44.079783 20873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:44.086444 20873 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43983
I20260619 18:41:44.086459 20940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43983 every 8 connection(s)
I20260619 18:41:44.087342 20873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:44.090101 20941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:44.091162 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 20873
I20260619 18:41:44.091305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:44.091615 392 external_mini_cluster.cc:1428] Setting key 597bfd7da1573d2c23fd15b6f1228877
I20260619 18:41:44.096402 20941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76: Bootstrap starting.
I20260619 18:41:44.096741 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:44.092948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43883 (local address 127.0.0.1:43983)
0619 18:41:44.093373 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:44.093391 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:44.093426 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:41:44.094090 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:44.094108 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:44.094415 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:44.094703 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:44.095539 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:44.095543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:44.095555 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:44.095565 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:44.095616 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:44.095634 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:44.095743 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:44.095838 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 18:41:44.095979 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":107,"threads_started":1}
I20260619 18:41:44.098773 20941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:44.099411 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:41:44.099555 20941 log.cc:824] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:44.101688 20941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76: No bootstrap required, opened a new log
I20260619 18:41:44.103061 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:44.101497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51277 (local address 127.0.0.1:43983)
0619 18:41:44.101619 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:44.101631 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:44.101806 (+ 175us) server_negotiation.cc:407] Connection header received
0619 18:41:44.101902 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:44.101927 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:44.101962 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:44.102045 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:44.102731 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:44.102734 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:44.102737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:44.102748 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:44.102773 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:44.102785 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:44.102868 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:44.102913 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 18:41:44.102947 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 18:41:44.105261 20941 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3532e301e2f4d90810aa1c2b2236c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } }
I20260619 18:41:44.105472 20941 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:44.105504 20941 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3532e301e2f4d90810aa1c2b2236c76, State: Initialized, Role: FOLLOWER
I20260619 18:41:44.105849 20941 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3532e301e2f4d90810aa1c2b2236c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } }
I20260619 18:41:44.105993 20941 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:44.106042 20941 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:44.106134 20941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:44.106971 20941 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3532e301e2f4d90810aa1c2b2236c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } }
I20260619 18:41:44.107292 20941 leader_election.cc:302] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3532e301e2f4d90810aa1c2b2236c76; no voters:
I20260619 18:41:44.107589 20941 leader_election.cc:288] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:44.107709 20951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:44.107926 20951 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [term 1 LEADER]: Becoming Leader. State: Replica: e3532e301e2f4d90810aa1c2b2236c76, State: Running, Role: LEADER
I20260619 18:41:44.108287 20951 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3532e301e2f4d90810aa1c2b2236c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } }
I20260619 18:41:44.108744 20941 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:44.110028 20952 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3532e301e2f4d90810aa1c2b2236c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3532e301e2f4d90810aa1c2b2236c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } } }
I20260619 18:41:44.110145 20952 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:44.110037 20953 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3532e301e2f4d90810aa1c2b2236c76. Latest consensus state: current_term: 1 leader_uuid: "e3532e301e2f4d90810aa1c2b2236c76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3532e301e2f4d90810aa1c2b2236c76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43983 } } }
I20260619 18:41:44.110512 20953 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76 [sys.catalog]: This master's current role is: LEADER
W20260619 18:41:44.113292 20966 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:41:44.113384 20966 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:41:44.113479 20960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:44.114423 20960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:44.119973 20960 catalog_manager.cc:1347] Generated new cluster ID: 326365c32937404bbd51d2b719c146a0
I20260619 18:41:44.120033 20960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:44.144140 20960 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:44.144517 20960 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:44.152220 20960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3532e301e2f4d90810aa1c2b2236c76: Generated new TSK 0
I20260619 18:41:44.152935 20960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:44.155856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 20873
2026-06-19T18:41:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:44 dist-test-slave-rxv6 krb5kdc[20972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:44 dist-test-slave-rxv6 krb5kdc[20972](info): set up 2 sockets
Jun 19 18:41:44 dist-test-slave-rxv6 krb5kdc[20972](info): commencing operation
krb5kdc: starting...
W20260619 18:41:45.117811 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:45 dist-test-slave-rxv6 krb5kdc[20972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:41:45.230345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:45.337888 20985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:45.338186 20985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:45.338248 20985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:45.342310 20985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:45.342387 20985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:45.342413 20985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:45.342437 20985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:45.342464 20985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:45.347818 20985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:45.348903 20985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:45.349946 20985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:45.358537 20991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:45.358553 20993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:45.358546 20990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:45.358597 20985 server_base.cc:1034] running on GCE node
I20260619 18:41:45.359349 20985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:45.360517 20985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:45.361702 20985 hybrid_clock.cc:648] HybridClock initialized: now 1781894505361685 us; error 38 us; skew 500 ppm
Jun 19 18:41:45 dist-test-slave-rxv6 krb5kdc[20972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:45.364639 20985 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:41:45.365685 20985 webserver.cc:460] Webserver started at http://127.0.0.1:35119/ using document root <none> and password file <none>
I20260619 18:41:45.366309 20985 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:45.366379 20985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:45.366572 20985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:45.368775 20985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "48bf7b2425354d1ca23d9dd0fe30f4a9"
format_stamp: "Formatted at 2026-06-19 18:41:45 on dist-test-slave-rxv6"
server_key: "d4dba38ca6144f4df45eb268ecbcf7f6"
server_key_iv: "5ac2c15150c4bc8c4e02eda02a363b34"
server_key_version: "encryptionkey@0"
I20260619 18:41:45.369472 20985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "48bf7b2425354d1ca23d9dd0fe30f4a9"
format_stamp: "Formatted at 2026-06-19 18:41:45 on dist-test-slave-rxv6"
server_key: "d4dba38ca6144f4df45eb268ecbcf7f6"
server_key_iv: "5ac2c15150c4bc8c4e02eda02a363b34"
server_key_version: "encryptionkey@0"
I20260619 18:41:45.373533 20985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 18:41:45.376045 21000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:45.377274 20985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:41:45.377442 20985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "48bf7b2425354d1ca23d9dd0fe30f4a9"
format_stamp: "Formatted at 2026-06-19 18:41:45 on dist-test-slave-rxv6"
server_key: "d4dba38ca6144f4df45eb268ecbcf7f6"
server_key_iv: "5ac2c15150c4bc8c4e02eda02a363b34"
server_key_version: "encryptionkey@0"
I20260619 18:41:45.377549 20985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:45.390475 20985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:45.393682 20985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:45.393872 20985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:45.400841 20985 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35197
I20260619 18:41:45.400856 21052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35197 every 8 connection(s)
I20260619 18:41:45.401798 20985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:45.404628 21053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:45.406232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 20985
I20260619 18:41:45.406349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:45.406603 392 external_mini_cluster.cc:1428] Setting key fef189a68c3e6567de749842c696dddc
I20260619 18:41:45.411401 21053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9: Bootstrap starting.
I20260619 18:41:45.414028 21053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:45.414901 21053 log.cc:824] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:45.416904 21053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9: No bootstrap required, opened a new log
Jun 19 18:41:45 dist-test-slave-rxv6 krb5kdc[20972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:41:45.421025 21053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bf7b2425354d1ca23d9dd0fe30f4a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35197 } }
I20260619 18:41:45.421325 21053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:45.421393 21053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48bf7b2425354d1ca23d9dd0fe30f4a9, State: Initialized, Role: FOLLOWER
I20260619 18:41:45.421954 21053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bf7b2425354d1ca23d9dd0fe30f4a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35197 } }
I20260619 18:41:45.422123 21053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:45.422192 21053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:45.422299 21053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:45.423444 21053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bf7b2425354d1ca23d9dd0fe30f4a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35197 } }
I20260619 18:41:45.423804 21053 leader_election.cc:302] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48bf7b2425354d1ca23d9dd0fe30f4a9; no voters:
I20260619 18:41:45.424147 21053 leader_election.cc:288] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:45.424239 21058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:45.424461 21058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [term 1 LEADER]: Becoming Leader. State: Replica: 48bf7b2425354d1ca23d9dd0fe30f4a9, State: Running, Role: LEADER
I20260619 18:41:45.424813 21058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bf7b2425354d1ca23d9dd0fe30f4a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35197 } }
I20260619 18:41:45.425457 21053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:45.426497 21060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48bf7b2425354d1ca23d9dd0fe30f4a9. Latest consensus state: current_term: 1 leader_uuid: "48bf7b2425354d1ca23d9dd0fe30f4a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bf7b2425354d1ca23d9dd0fe30f4a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35197 } } }
I20260619 18:41:45.426609 21060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:45.426709 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:45.408043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37073 (local address 127.0.0.1:35197)
0619 18:41:45.408479 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:45.408498 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:45.408542 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:41:45.409409 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:45.409434 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:45.409885 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:45.410300 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:45.411003 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.412437 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.413397 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.414122 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.414297 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:45.421062 (+ 6765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:45.421069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:45.421087 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:45.421125 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:45.423132 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:45.423596 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:45.423602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:45.423608 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:45.423749 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:45.424033 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:45.424036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:45.424037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:45.424568 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:45.424752 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:45.425395 (+ 643us) server_negotiation.cc:299] Negotiation successful
0619 18:41:45.425693 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":97,"threads_started":1}
I20260619 18:41:45.427799 21059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48bf7b2425354d1ca23d9dd0fe30f4a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48bf7b2425354d1ca23d9dd0fe30f4a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35197 } } }
I20260619 18:41:45.427894 21059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:45.429854 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 18:41:45.430809 21073 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:41:45.430898 21073 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:41:45.431281 21067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:45.432101 21067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:45.438134 21067 catalog_manager.cc:1347] Generated new cluster ID: 73ae97d4fe854e56aa93d0ddddbdb500
I20260619 18:41:45.438199 21067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:45.444281 21056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:45.434336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51915 (local address 127.0.0.1:35197)
0619 18:41:45.434439 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:45.434444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:45.434703 (+ 259us) server_negotiation.cc:407] Connection header received
0619 18:41:45.434843 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:45.434846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:45.434893 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:45.434977 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:45.435788 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.436406 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.437223 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:45.437475 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:45.437566 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:45.441123 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:45.441126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:45.441129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:45.441155 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:45.442742 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:45.443183 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:45.443186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:45.443188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:45.443260 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:45.443498 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:45.443500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:45.443502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:45.443687 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:45.443799 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:45.444008 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 18:41:45.444122 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 18:41:45.451856 21067 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:45.453145 21067 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:45.461076 21067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48bf7b2425354d1ca23d9dd0fe30f4a9: Generated new TSK 0
I20260619 18:41:45.461774 21067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:45.468072 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 20985
2026-06-19T18:41:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:45 dist-test-slave-rxv6 krb5kdc[21084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:45 dist-test-slave-rxv6 krb5kdc[21084](info): set up 2 sockets
Jun 19 18:41:45 dist-test-slave-rxv6 krb5kdc[21084](info): commencing operation
krb5kdc: starting...
W20260619 18:41:46.434021 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:46 dist-test-slave-rxv6 krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:41:46.553545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40565
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:46.663780 21097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:46.664052 21097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:46.664093 21097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:46.668143 21097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:46.668221 21097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:46.668239 21097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:46.668256 21097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:46.668269 21097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:46.673735 21097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:46.674803 21097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:46.675752 21097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:46.684093 21102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:46.684100 21105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:46.684104 21103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:46.684617 21097 server_base.cc:1034] running on GCE node
I20260619 18:41:46.685158 21097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:46.686371 21097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:46.687572 21097 hybrid_clock.cc:648] HybridClock initialized: now 1781894506687553 us; error 34 us; skew 500 ppm
Jun 19 18:41:46 dist-test-slave-rxv6 krb5kdc[21084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:46.697852 21097 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:41:46.698948 21097 webserver.cc:460] Webserver started at http://127.0.0.1:33697/ using document root <none> and password file <none>
I20260619 18:41:46.699527 21097 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:46.699589 21097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:46.699754 21097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:46.701750 21097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "f6cef2e5f3a6498880f7b79486115f56"
format_stamp: "Formatted at 2026-06-19 18:41:46 on dist-test-slave-rxv6"
server_key: "d6342a6c308d460241ece51cf008f9eb"
server_key_iv: "e6d87922eba4df4e6387212a9a5f385c"
server_key_version: "encryptionkey@0"
I20260619 18:41:46.702301 21097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "f6cef2e5f3a6498880f7b79486115f56"
format_stamp: "Formatted at 2026-06-19 18:41:46 on dist-test-slave-rxv6"
server_key: "d6342a6c308d460241ece51cf008f9eb"
server_key_iv: "e6d87922eba4df4e6387212a9a5f385c"
server_key_version: "encryptionkey@0"
I20260619 18:41:46.706012 21097 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:41:46.708400 21112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:46.709776 21097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:41:46.709939 21097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "f6cef2e5f3a6498880f7b79486115f56"
format_stamp: "Formatted at 2026-06-19 18:41:46 on dist-test-slave-rxv6"
server_key: "d6342a6c308d460241ece51cf008f9eb"
server_key_iv: "e6d87922eba4df4e6387212a9a5f385c"
server_key_version: "encryptionkey@0"
I20260619 18:41:46.710064 21097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:46.727598 21097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:46.730636 21097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:46.730813 21097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:46.737422 21097 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40565
I20260619 18:41:46.737438 21164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40565 every 8 connection(s)
I20260619 18:41:46.738389 21097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:46.739830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 21097
I20260619 18:41:46.740012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:46.740284 392 external_mini_cluster.cc:1428] Setting key fc1e00461aa76c286bc6cf36da22d3c1
I20260619 18:41:46.741374 21165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:46.748380 21165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56: Bootstrap starting.
I20260619 18:41:46.750650 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:46.741616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48591 (local address 127.0.0.1:40565)
0619 18:41:46.742081 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:46.742100 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:46.742144 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:41:46.743043 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:46.743067 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:46.743517 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:46.743901 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:46.744327 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.745806 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.746818 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.747537 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.747694 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:46.748551 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:46.748557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:46.748574 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:46.749082 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:46.749150 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:46.749180 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:46.749274 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:46.749429 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 18:41:46.749655 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":121,"threads_started":1}
I20260619 18:41:46.751194 21165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:46.752028 21165 log.cc:824] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:46.753690 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:41:46.754236 21165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56: No bootstrap required, opened a new log
I20260619 18:41:46.758327 21165 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cef2e5f3a6498880f7b79486115f56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40565 } }
I20260619 18:41:46.758585 21165 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:46.758668 21165 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6cef2e5f3a6498880f7b79486115f56, State: Initialized, Role: FOLLOWER
I20260619 18:41:46.759150 21165 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cef2e5f3a6498880f7b79486115f56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40565 } }
I20260619 18:41:46.759310 21165 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:46.759398 21165 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:46.759518 21165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:46.759837 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:46.755664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58309 (local address 127.0.0.1:40565)
0619 18:41:46.755775 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:46.755779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:46.755933 (+ 154us) server_negotiation.cc:407] Connection header received
0619 18:41:46.756079 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:46.756082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:46.756144 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:46.756234 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:46.756596 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.757352 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.758223 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:46.758466 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:46.758554 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:46.759416 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:46.759421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:46.759424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:46.759439 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:46.759471 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:46.759486 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:46.759564 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:46.759652 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 18:41:46.759711 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:41:46.760533 21165 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cef2e5f3a6498880f7b79486115f56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40565 } }
I20260619 18:41:46.760841 21165 leader_election.cc:302] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6cef2e5f3a6498880f7b79486115f56; no voters:
I20260619 18:41:46.761124 21165 leader_election.cc:288] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:46.761270 21175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:46.761570 21175 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [term 1 LEADER]: Becoming Leader. State: Replica: f6cef2e5f3a6498880f7b79486115f56, State: Running, Role: LEADER
I20260619 18:41:46.761891 21175 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cef2e5f3a6498880f7b79486115f56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40565 } }
I20260619 18:41:46.762328 21165 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:46.763646 21176 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6cef2e5f3a6498880f7b79486115f56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cef2e5f3a6498880f7b79486115f56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40565 } } }
I20260619 18:41:46.763682 21177 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6cef2e5f3a6498880f7b79486115f56. Latest consensus state: current_term: 1 leader_uuid: "f6cef2e5f3a6498880f7b79486115f56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6cef2e5f3a6498880f7b79486115f56" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40565 } } }
I20260619 18:41:46.763857 21176 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:46.763883 21177 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:46.764256 21188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:46.767745 21188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:46.773298 21188 catalog_manager.cc:1347] Generated new cluster ID: 68a7aab65cb540e38c51397d788e251e
I20260619 18:41:46.773437 21188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:46.788393 21188 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:46.789350 21188 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:46.795171 21188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6cef2e5f3a6498880f7b79486115f56: Generated new TSK 0
I20260619 18:41:46.795847 21188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:46.802227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 21097
2026-06-19T18:41:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:46 dist-test-slave-rxv6 krb5kdc[21196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:46 dist-test-slave-rxv6 krb5kdc[21196](info): set up 2 sockets
Jun 19 18:41:46 dist-test-slave-rxv6 krb5kdc[21196](info): commencing operation
krb5kdc: starting...
W20260619 18:41:47.761878 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:47 dist-test-slave-rxv6 krb5kdc[21196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:41:47.880941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40761
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:47.989704 21209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:47.990027 21209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:47.990084 21209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:47.994184 21209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:47.994251 21209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:47.994272 21209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:47.994288 21209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:47.994302 21209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:47.999730 21209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:48.000793 21209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:48.001791 21209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:41:48.010497 21209 server_base.cc:1034] running on GCE node
W20260619 18:41:48.010439 21217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:48.010445 21214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:48.010447 21215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:48.011246 21209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:48.012528 21209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:48.013706 21209 hybrid_clock.cc:648] HybridClock initialized: now 1781894508013691 us; error 29 us; skew 500 ppm
Jun 19 18:41:48 dist-test-slave-rxv6 krb5kdc[21196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:48.016474 21209 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:41:48.017366 21209 webserver.cc:460] Webserver started at http://127.0.0.1:38683/ using document root <none> and password file <none>
I20260619 18:41:48.017874 21209 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:48.017966 21209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:48.018157 21209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:48.020148 21209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "daa1d20f098d44de9e1b6d2a917e8da1"
format_stamp: "Formatted at 2026-06-19 18:41:48 on dist-test-slave-rxv6"
server_key: "dd9364d18d466928f019e1e904ae2430"
server_key_iv: "ed59d6ade80a99943cdb0c2923c2aeb7"
server_key_version: "encryptionkey@0"
I20260619 18:41:48.020654 21209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "daa1d20f098d44de9e1b6d2a917e8da1"
format_stamp: "Formatted at 2026-06-19 18:41:48 on dist-test-slave-rxv6"
server_key: "dd9364d18d466928f019e1e904ae2430"
server_key_iv: "ed59d6ade80a99943cdb0c2923c2aeb7"
server_key_version: "encryptionkey@0"
I20260619 18:41:48.024222 21209 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 18:41:48.026504 21224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:48.027707 21209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:41:48.027807 21209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "daa1d20f098d44de9e1b6d2a917e8da1"
format_stamp: "Formatted at 2026-06-19 18:41:48 on dist-test-slave-rxv6"
server_key: "dd9364d18d466928f019e1e904ae2430"
server_key_iv: "ed59d6ade80a99943cdb0c2923c2aeb7"
server_key_version: "encryptionkey@0"
I20260619 18:41:48.027891 21209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:48.032305 21209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:48.035324 21209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:48.035468 21209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:48.041898 21209 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40761
I20260619 18:41:48.041976 21276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40761 every 8 connection(s)
I20260619 18:41:48.042888 21209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:48.045782 21277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:48.046831 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 21209
I20260619 18:41:48.046947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:48.047214 392 external_mini_cluster.cc:1428] Setting key f7b94efba76c4302da33cbc32e840e1a
I20260619 18:41:48.052398 21277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1: Bootstrap starting.
I20260619 18:41:48.052664 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:48.048580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38335 (local address 127.0.0.1:40761)
0619 18:41:48.049011 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:48.049035 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:48.049071 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:41:48.049706 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:48.049723 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:48.050068 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:48.050359 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:48.051204 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:48.051208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:48.051220 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:48.051231 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:48.051289 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:48.051309 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:48.051434 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:48.051698 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 18:41:48.051875 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":121,"threads_started":1}
I20260619 18:41:48.054952 21277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:48.055570 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:41:48.055719 21277 log.cc:824] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:48.057870 21277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1: No bootstrap required, opened a new log
I20260619 18:41:48.059167 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:48.057595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44399 (local address 127.0.0.1:40761)
0619 18:41:48.057704 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:48.057708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:48.057940 (+ 232us) server_negotiation.cc:407] Connection header received
0619 18:41:48.058058 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:48.058061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:48.058095 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:48.058158 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:48.058818 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:48.058822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:48.058824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:48.058836 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:48.058864 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:48.058875 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:48.058944 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:48.059016 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 18:41:48.059053 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 18:41:48.061698 21277 raft_consensus.cc:348] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa1d20f098d44de9e1b6d2a917e8da1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40761 } }
I20260619 18:41:48.062026 21277 raft_consensus.cc:374] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:48.062088 21277 raft_consensus.cc:729] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa1d20f098d44de9e1b6d2a917e8da1, State: Initialized, Role: FOLLOWER
I20260619 18:41:48.062547 21277 consensus_queue.cc:260] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa1d20f098d44de9e1b6d2a917e8da1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40761 } }
I20260619 18:41:48.062680 21277 raft_consensus.cc:388] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:48.062737 21277 raft_consensus.cc:482] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:48.062834 21277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:48.063759 21277 raft_consensus.cc:504] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa1d20f098d44de9e1b6d2a917e8da1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40761 } }
I20260619 18:41:48.064085 21277 leader_election.cc:302] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daa1d20f098d44de9e1b6d2a917e8da1; no voters:
I20260619 18:41:48.064394 21277 leader_election.cc:288] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:48.064496 21287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:48.064764 21287 raft_consensus.cc:686] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [term 1 LEADER]: Becoming Leader. State: Replica: daa1d20f098d44de9e1b6d2a917e8da1, State: Running, Role: LEADER
I20260619 18:41:48.065068 21287 consensus_queue.cc:237] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa1d20f098d44de9e1b6d2a917e8da1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40761 } }
I20260619 18:41:48.065646 21277 sys_catalog.cc:564] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:48.066669 21288 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daa1d20f098d44de9e1b6d2a917e8da1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa1d20f098d44de9e1b6d2a917e8da1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40761 } } }
I20260619 18:41:48.067054 21288 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:48.066942 21289 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daa1d20f098d44de9e1b6d2a917e8da1. Latest consensus state: current_term: 1 leader_uuid: "daa1d20f098d44de9e1b6d2a917e8da1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa1d20f098d44de9e1b6d2a917e8da1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40761 } } }
I20260619 18:41:48.067440 21289 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:48.067534 21296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:48.070382 21296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:48.076364 21296 catalog_manager.cc:1347] Generated new cluster ID: 76d71a5907074f0386e585e5b8d3c06e
I20260619 18:41:48.076442 21296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:48.087338 21296 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:48.087695 21296 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:48.100315 21296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daa1d20f098d44de9e1b6d2a917e8da1: Generated new TSK 0
I20260619 18:41:48.101091 21296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:48.112720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 21209
2026-06-19T18:41:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:48 dist-test-slave-rxv6 krb5kdc[21308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:48 dist-test-slave-rxv6 krb5kdc[21308](info): set up 2 sockets
Jun 19 18:41:48 dist-test-slave-rxv6 krb5kdc[21308](info): commencing operation
krb5kdc: starting...
W20260619 18:41:49.074002 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:49 dist-test-slave-rxv6 krb5kdc[21308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:41:49.193060 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:49.303264 21321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:49.303560 21321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:49.303622 21321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:49.307646 21321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:49.307718 21321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:49.307744 21321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:49.307765 21321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:49.307785 21321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:49.313094 21321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:49.314199 21321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:49.315207 21321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:49.323570 21329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:49.323529 21327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:49.323532 21326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:49.323856 21321 server_base.cc:1034] running on GCE node
I20260619 18:41:49.324370 21321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:49.325487 21321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:49.326665 21321 hybrid_clock.cc:648] HybridClock initialized: now 1781894509326648 us; error 34 us; skew 500 ppm
Jun 19 18:41:49 dist-test-slave-rxv6 krb5kdc[21308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:49.329473 21321 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:41:49.330470 21321 webserver.cc:460] Webserver started at http://127.0.0.1:45805/ using document root <none> and password file <none>
I20260619 18:41:49.331038 21321 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:49.331099 21321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:49.331274 21321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:49.333431 21321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "177c076689b54653bacc4304b37c42d8"
format_stamp: "Formatted at 2026-06-19 18:41:49 on dist-test-slave-rxv6"
server_key: "8824f5eb8fdffe1a7c8ac3c9fbba16dd"
server_key_iv: "d49f138cc767adcbe395487c9266b15c"
server_key_version: "encryptionkey@0"
I20260619 18:41:49.334087 21321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "177c076689b54653bacc4304b37c42d8"
format_stamp: "Formatted at 2026-06-19 18:41:49 on dist-test-slave-rxv6"
server_key: "8824f5eb8fdffe1a7c8ac3c9fbba16dd"
server_key_iv: "d49f138cc767adcbe395487c9266b15c"
server_key_version: "encryptionkey@0"
I20260619 18:41:49.338040 21321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 18:41:49.340509 21336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:49.341729 21321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:41:49.341858 21321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "177c076689b54653bacc4304b37c42d8"
format_stamp: "Formatted at 2026-06-19 18:41:49 on dist-test-slave-rxv6"
server_key: "8824f5eb8fdffe1a7c8ac3c9fbba16dd"
server_key_iv: "d49f138cc767adcbe395487c9266b15c"
server_key_version: "encryptionkey@0"
I20260619 18:41:49.342021 21321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:49.362269 21321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:49.365535 21321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:49.365715 21321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:49.372979 21321 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44137
I20260619 18:41:49.373003 21388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44137 every 8 connection(s)
I20260619 18:41:49.374022 21321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:49.376804 21389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:49.379513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 21321
I20260619 18:41:49.379653 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:49.379902 392 external_mini_cluster.cc:1428] Setting key a20edfc1a5f5d43056a0e9e3d1903cf7
I20260619 18:41:49.382762 21389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8: Bootstrap starting.
I20260619 18:41:49.385046 21389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:49.385754 21389 log.cc:824] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:49.387590 21389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8: No bootstrap required, opened a new log
Jun 19 18:41:49 dist-test-slave-rxv6 krb5kdc[21308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:41:49.391037 21389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c076689b54653bacc4304b37c42d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44137 } }
I20260619 18:41:49.391271 21389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:49.391335 21389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 177c076689b54653bacc4304b37c42d8, State: Initialized, Role: FOLLOWER
I20260619 18:41:49.391835 21389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c076689b54653bacc4304b37c42d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44137 } }
I20260619 18:41:49.392011 21389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:49.392087 21389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:49.392185 21389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:49.393187 21389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c076689b54653bacc4304b37c42d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44137 } }
I20260619 18:41:49.393532 21389 leader_election.cc:302] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 177c076689b54653bacc4304b37c42d8; no voters:
I20260619 18:41:49.393707 21392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:49.381289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:35491 (local address 127.0.0.1:44137)
0619 18:41:49.381764 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:49.381782 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:49.381826 (+ 44us) server_negotiation.cc:407] Connection header received
0619 18:41:49.382485 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:49.382501 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:49.382924 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:49.383241 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:49.384016 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:49.385021 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:49.385898 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:49.386414 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:49.386561 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:49.388872 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:49.388878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:49.388891 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:49.388917 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:49.390488 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:49.390958 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:49.390962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:49.390967 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:49.391070 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:49.391409 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:49.391413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:49.391415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:49.391938 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:49.392157 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:49.392645 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 18:41:49.392883 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":159,"threads_started":1}
I20260619 18:41:49.393833 21389 leader_election.cc:288] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:49.394001 21394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:49.394253 21394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [term 1 LEADER]: Becoming Leader. State: Replica: 177c076689b54653bacc4304b37c42d8, State: Running, Role: LEADER
I20260619 18:41:49.394548 21394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c076689b54653bacc4304b37c42d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44137 } }
I20260619 18:41:49.395149 21389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:49.395711 21396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 177c076689b54653bacc4304b37c42d8. Latest consensus state: current_term: 1 leader_uuid: "177c076689b54653bacc4304b37c42d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c076689b54653bacc4304b37c42d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44137 } } }
I20260619 18:41:49.395838 21396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:49.396466 21395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "177c076689b54653bacc4304b37c42d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "177c076689b54653bacc4304b37c42d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44137 } } }
I20260619 18:41:49.396552 21395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:49.396912 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:41:49.397966 21400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:49.401218 21400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:49.407270 21400 catalog_manager.cc:1347] Generated new cluster ID: 61766d0b9ecc43d6a086d83e1d9cfaf5
I20260619 18:41:49.407433 21400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:49.407943 21392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:49.399115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:33353 (local address 127.0.0.1:44137)
0619 18:41:49.399220 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:49.399224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:49.399502 (+ 278us) server_negotiation.cc:407] Connection header received
0619 18:41:49.399615 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:49.399618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:49.399668 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:49.399756 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:49.400521 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:49.401264 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:49.402377 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:49.402749 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:49.402854 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:49.404284 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:49.404287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:49.404290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:49.404315 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:49.406085 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:49.406524 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:49.406528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:49.406529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:49.406604 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:49.406898 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:49.406902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:49.406903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:49.407118 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:49.407248 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:49.407681 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 18:41:49.407794 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 18:41:49.423804 21400 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:49.425099 21400 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:49.434780 21400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 177c076689b54653bacc4304b37c42d8: Generated new TSK 0
I20260619 18:41:49.435675 21400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:49.442759 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 21321
W20260619 18:41:49.447580 21306 connection.cc:535] client connection to 127.0.0.1:44137 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44137): Connection reset by peer (error 104)
2026-06-19T18:41:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:49 dist-test-slave-rxv6 krb5kdc[21420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:49 dist-test-slave-rxv6 krb5kdc[21420](info): set up 2 sockets
Jun 19 18:41:49 dist-test-slave-rxv6 krb5kdc[21420](info): commencing operation
krb5kdc: starting...
W20260619 18:41:50.401865 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:50 dist-test-slave-rxv6 krb5kdc[21420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:41:50.518545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41693
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:50.626031 21433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:50.626292 21433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:50.626333 21433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:50.630328 21433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:50.630385 21433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:50.630405 21433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:50.630435 21433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:50.630461 21433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:50.635699 21433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:50.636682 21433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:50.637663 21433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:50.645802 21439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:50.645830 21441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:50.645809 21438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:50.645804 21433 server_base.cc:1034] running on GCE node
I20260619 18:41:50.646541 21433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:50.647603 21433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:50.648753 21433 hybrid_clock.cc:648] HybridClock initialized: now 1781894510648741 us; error 34 us; skew 500 ppm
Jun 19 18:41:50 dist-test-slave-rxv6 krb5kdc[21420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:50.651330 21433 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:41:50.652141 21433 webserver.cc:460] Webserver started at http://127.0.0.1:41175/ using document root <none> and password file <none>
I20260619 18:41:50.652654 21433 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:50.652709 21433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:50.652864 21433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:50.654896 21433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "8281a1e58b0646459d48b96b35bcd97f"
format_stamp: "Formatted at 2026-06-19 18:41:50 on dist-test-slave-rxv6"
server_key: "70f1bed91b75d51cea3b0176fafcd2d0"
server_key_iv: "51ef9d7dc3ae100fd04f1bcfd72a75cb"
server_key_version: "encryptionkey@0"
I20260619 18:41:50.655377 21433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "8281a1e58b0646459d48b96b35bcd97f"
format_stamp: "Formatted at 2026-06-19 18:41:50 on dist-test-slave-rxv6"
server_key: "70f1bed91b75d51cea3b0176fafcd2d0"
server_key_iv: "51ef9d7dc3ae100fd04f1bcfd72a75cb"
server_key_version: "encryptionkey@0"
I20260619 18:41:50.658916 21433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:41:50.661175 21448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:50.662554 21433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:41:50.662667 21433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "8281a1e58b0646459d48b96b35bcd97f"
format_stamp: "Formatted at 2026-06-19 18:41:50 on dist-test-slave-rxv6"
server_key: "70f1bed91b75d51cea3b0176fafcd2d0"
server_key_iv: "51ef9d7dc3ae100fd04f1bcfd72a75cb"
server_key_version: "encryptionkey@0"
I20260619 18:41:50.662761 21433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:50.684576 21433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:50.687645 21433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:50.687799 21433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:50.694458 21433 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41693
I20260619 18:41:50.694568 21500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41693 every 8 connection(s)
I20260619 18:41:50.695438 21433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:50.698130 21501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:50.703378 21501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f: Bootstrap starting.
I20260619 18:41:50.704519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 21433
I20260619 18:41:50.704649 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:50.704909 392 external_mini_cluster.cc:1428] Setting key 5adb94f3315fff36c0112b5cd0d6f8fa
I20260619 18:41:50.706113 21501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:50.706826 21501 log.cc:824] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:50.708711 21501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f: No bootstrap required, opened a new log
I20260619 18:41:50.712666 21501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8281a1e58b0646459d48b96b35bcd97f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } }
I20260619 18:41:50.712889 21501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:50.712952 21501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8281a1e58b0646459d48b96b35bcd97f, State: Initialized, Role: FOLLOWER
I20260619 18:41:50.713286 21504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:50.706019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:56317 (local address 127.0.0.1:41693)
0619 18:41:50.706439 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:50.706455 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:50.706489 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:41:50.707128 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:50.707146 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:50.707483 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:50.707753 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:50.708054 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:50.709025 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:50.710199 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:50.710648 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:50.710758 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:50.711615 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:50.711620 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:50.711632 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:50.712002 (+ 370us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:50.712053 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:50.712074 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:50.712170 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:50.712337 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 18:41:50.712500 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":122,"threads_started":1}
I20260619 18:41:50.713351 21501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8281a1e58b0646459d48b96b35bcd97f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } }
I20260619 18:41:50.713523 21501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:50.713591 21501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:50.713685 21501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:50.714684 21501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8281a1e58b0646459d48b96b35bcd97f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } }
I20260619 18:41:50.714990 21501 leader_election.cc:302] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8281a1e58b0646459d48b96b35bcd97f; no voters:
I20260619 18:41:50.715297 21501 leader_election.cc:288] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:50.715391 21506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:50.715591 21506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [term 1 LEADER]: Becoming Leader. State: Replica: 8281a1e58b0646459d48b96b35bcd97f, State: Running, Role: LEADER
I20260619 18:41:50.715916 21506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8281a1e58b0646459d48b96b35bcd97f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } }
I20260619 18:41:50.716239 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:41:50.716547 21501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:50.717063 21508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8281a1e58b0646459d48b96b35bcd97f. Latest consensus state: current_term: 1 leader_uuid: "8281a1e58b0646459d48b96b35bcd97f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8281a1e58b0646459d48b96b35bcd97f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } } }
I20260619 18:41:50.717175 21508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:50.717787 21507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8281a1e58b0646459d48b96b35bcd97f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8281a1e58b0646459d48b96b35bcd97f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41693 } } }
I20260619 18:41:50.717871 21507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:50.718781 21512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:50.721681 21512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:50.722445 21504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:50.718543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:46045 (local address 127.0.0.1:41693)
0619 18:41:50.718760 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:50.718764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:50.718837 (+ 73us) server_negotiation.cc:407] Connection header received
0619 18:41:50.719247 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:50.719250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:50.719295 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:50.719368 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:50.719641 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:50.720146 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:50.720920 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:50.721148 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:50.721231 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:50.722032 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:50.722036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:50.722039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:50.722051 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:50.722076 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:50.722087 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:50.722158 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:50.722275 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 18:41:50.722304 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260619 18:41:50.727245 21512 catalog_manager.cc:1347] Generated new cluster ID: c9cc4991c5e24dc19c568db3b2c3a13d
I20260619 18:41:50.727304 21512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:50.734673 21512 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:50.735512 21512 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:50.741894 21512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8281a1e58b0646459d48b96b35bcd97f: Generated new TSK 0
I20260619 18:41:50.742530 21512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:50.747938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 21433
W20260619 18:41:50.752628 21418 connection.cc:535] client connection to 127.0.0.1:41693 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41693): Connection reset by peer (error 104)
2026-06-19T18:41:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:50 dist-test-slave-rxv6 krb5kdc[21532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:50 dist-test-slave-rxv6 krb5kdc[21532](info): set up 2 sockets
Jun 19 18:41:50 dist-test-slave-rxv6 krb5kdc[21532](info): commencing operation
krb5kdc: starting...
W20260619 18:41:51.709900 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:51 dist-test-slave-rxv6 krb5kdc[21532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:41:51.828574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33035
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:51.936188 21545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:51.936467 21545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:51.936525 21545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:51.940564 21545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:51.940634 21545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:51.940662 21545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:51.940682 21545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:51.940703 21545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:51.946036 21545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:51.947079 21545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:51.948079 21545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:51.956467 21550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:51.956467 21553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:51.956461 21551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:51.956727 21545 server_base.cc:1034] running on GCE node
I20260619 18:41:51.957227 21545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:51.958350 21545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:51.959511 21545 hybrid_clock.cc:648] HybridClock initialized: now 1781894511959497 us; error 31 us; skew 500 ppm
Jun 19 18:41:51 dist-test-slave-rxv6 krb5kdc[21532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:51.966641 21545 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:41:51.967577 21545 webserver.cc:460] Webserver started at http://127.0.0.1:43147/ using document root <none> and password file <none>
I20260619 18:41:51.968142 21545 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:51.968211 21545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:51.968417 21545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:51.970577 21545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "c0e34f7f0f9b4c61afb91d50e9ae5d5a"
format_stamp: "Formatted at 2026-06-19 18:41:51 on dist-test-slave-rxv6"
server_key: "5868743c46fbd6f8af17830bd007d9a7"
server_key_iv: "75aed17f5318067c7e3ff4da148230df"
server_key_version: "encryptionkey@0"
I20260619 18:41:51.971120 21545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "c0e34f7f0f9b4c61afb91d50e9ae5d5a"
format_stamp: "Formatted at 2026-06-19 18:41:51 on dist-test-slave-rxv6"
server_key: "5868743c46fbd6f8af17830bd007d9a7"
server_key_iv: "75aed17f5318067c7e3ff4da148230df"
server_key_version: "encryptionkey@0"
I20260619 18:41:51.975456 21545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 18:41:51.978192 21560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:51.979600 21545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:41:51.979733 21545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "c0e34f7f0f9b4c61afb91d50e9ae5d5a"
format_stamp: "Formatted at 2026-06-19 18:41:51 on dist-test-slave-rxv6"
server_key: "5868743c46fbd6f8af17830bd007d9a7"
server_key_iv: "75aed17f5318067c7e3ff4da148230df"
server_key_version: "encryptionkey@0"
I20260619 18:41:51.979856 21545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:51.984640 21545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:51.987807 21545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:51.987989 21545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:51.994984 21545 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33035
I20260619 18:41:51.995002 21612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33035 every 8 connection(s)
I20260619 18:41:51.995918 21545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:51.998728 21613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:52.004060 21613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a: Bootstrap starting.
I20260619 18:41:52.004443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 21545
I20260619 18:41:52.004577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:52.004861 392 external_mini_cluster.cc:1428] Setting key 72425e166cd1fcd2853da921fa2df38d
I20260619 18:41:52.006827 21613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:52.007547 21613 log.cc:824] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:52.009596 21613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a: No bootstrap required, opened a new log
I20260619 18:41:52.010026 21616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:52.006183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:54383 (local address 127.0.0.1:33035)
0619 18:41:52.006627 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:52.006641 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:52.006676 (+ 35us) server_negotiation.cc:407] Connection header received
0619 18:41:52.007342 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:52.007362 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:52.007685 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:52.008037 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:52.008765 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:52.008770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:52.008782 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:52.008793 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:52.008845 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:52.008864 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:52.008957 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:52.009080 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 18:41:52.009243 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":131,"threads_started":1}
I20260619 18:41:52.012614 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:41:52.013223 21613 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e34f7f0f9b4c61afb91d50e9ae5d5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } }
I20260619 18:41:52.013465 21613 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:52.013511 21613 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e34f7f0f9b4c61afb91d50e9ae5d5a, State: Initialized, Role: FOLLOWER
I20260619 18:41:52.013957 21613 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e34f7f0f9b4c61afb91d50e9ae5d5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } }
I20260619 18:41:52.014088 21613 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:52.014148 21613 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:52.014241 21613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:52.015383 21613 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e34f7f0f9b4c61afb91d50e9ae5d5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } }
I20260619 18:41:52.015720 21613 leader_election.cc:302] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0e34f7f0f9b4c61afb91d50e9ae5d5a; no voters:
I20260619 18:41:52.016010 21613 leader_election.cc:288] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:52.016103 21623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:52.016237 21616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:52.014579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:54037 (local address 127.0.0.1:33035)
0619 18:41:52.014685 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:52.014690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:52.014875 (+ 185us) server_negotiation.cc:407] Connection header received
0619 18:41:52.014998 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:52.015001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:52.015034 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:52.015096 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:52.015803 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:52.015807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:52.015809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:52.015821 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:52.015850 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:52.015863 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:52.015935 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:52.016038 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 18:41:52.016084 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 18:41:52.016321 21623 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [term 1 LEADER]: Becoming Leader. State: Replica: c0e34f7f0f9b4c61afb91d50e9ae5d5a, State: Running, Role: LEADER
I20260619 18:41:52.016633 21623 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e34f7f0f9b4c61afb91d50e9ae5d5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } }
I20260619 18:41:52.017319 21613 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:52.018345 21625 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0e34f7f0f9b4c61afb91d50e9ae5d5a. Latest consensus state: current_term: 1 leader_uuid: "c0e34f7f0f9b4c61afb91d50e9ae5d5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e34f7f0f9b4c61afb91d50e9ae5d5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } } }
I20260619 18:41:52.018373 21624 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0e34f7f0f9b4c61afb91d50e9ae5d5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0e34f7f0f9b4c61afb91d50e9ae5d5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33035 } } }
I20260619 18:41:52.018466 21625 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:52.018467 21624 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:52.018807 21632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:52.021677 21632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:52.026676 21632 catalog_manager.cc:1347] Generated new cluster ID: 524e338b70974a39bbcba79b0b19c031
I20260619 18:41:52.026751 21632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:52.039536 21632 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:52.039855 21632 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:52.046924 21632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0e34f7f0f9b4c61afb91d50e9ae5d5a: Generated new TSK 0
I20260619 18:41:52.047657 21632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:52.055768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 21545
2026-06-19T18:41:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:52 dist-test-slave-rxv6 krb5kdc[21644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:52 dist-test-slave-rxv6 krb5kdc[21644](info): set up 2 sockets
Jun 19 18:41:52 dist-test-slave-rxv6 krb5kdc[21644](info): commencing operation
krb5kdc: starting...
W20260619 18:41:53.005856 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:53 dist-test-slave-rxv6 krb5kdc[21644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:41:53.121559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38107
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:53.231501 21657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:53.231783 21657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:53.231825 21657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:53.235890 21657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:53.235966 21657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:53.235984 21657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:53.235999 21657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:53.236013 21657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:53.241431 21657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38107
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:53.242513 21657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:53.243479 21657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:53.252147 21663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:53.252156 21662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:53.252242 21657 server_base.cc:1034] running on GCE node
W20260619 18:41:53.252143 21665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:53.253024 21657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:53.254411 21657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:53.255595 21657 hybrid_clock.cc:648] HybridClock initialized: now 1781894513255570 us; error 41 us; skew 500 ppm
Jun 19 18:41:53 dist-test-slave-rxv6 krb5kdc[21644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:53.258658 21657 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:41:53.259595 21657 webserver.cc:460] Webserver started at http://127.0.0.1:39529/ using document root <none> and password file <none>
I20260619 18:41:53.260183 21657 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:53.260251 21657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:53.260453 21657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:53.262519 21657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "dd1cb3d687914da281d3bb50c12c0c08"
format_stamp: "Formatted at 2026-06-19 18:41:53 on dist-test-slave-rxv6"
server_key: "fca8f676779305194fcbb7b15fee5fe8"
server_key_iv: "0ecba06cb8e4dac016a9b364f3b7da10"
server_key_version: "encryptionkey@0"
I20260619 18:41:53.263077 21657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "dd1cb3d687914da281d3bb50c12c0c08"
format_stamp: "Formatted at 2026-06-19 18:41:53 on dist-test-slave-rxv6"
server_key: "fca8f676779305194fcbb7b15fee5fe8"
server_key_iv: "0ecba06cb8e4dac016a9b364f3b7da10"
server_key_version: "encryptionkey@0"
I20260619 18:41:53.266821 21657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 18:41:53.269362 21672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:53.270752 21657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 18:41:53.270884 21657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "dd1cb3d687914da281d3bb50c12c0c08"
format_stamp: "Formatted at 2026-06-19 18:41:53 on dist-test-slave-rxv6"
server_key: "fca8f676779305194fcbb7b15fee5fe8"
server_key_iv: "0ecba06cb8e4dac016a9b364f3b7da10"
server_key_version: "encryptionkey@0"
I20260619 18:41:53.270988 21657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:53.292320 21657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:53.295433 21657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:53.295624 21657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:53.302664 21657 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38107
I20260619 18:41:53.302687 21724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38107 every 8 connection(s)
I20260619 18:41:53.303596 21657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:53.306432 21725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:53.307806 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 21657
I20260619 18:41:53.307933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:53.308179 392 external_mini_cluster.cc:1428] Setting key d682dc5c5db92f3365e19d9b75c475c2
I20260619 18:41:53.312822 21725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08: Bootstrap starting.
I20260619 18:41:53.315434 21725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:53.316315 21725 log.cc:824] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:41:53 dist-test-slave-rxv6 krb5kdc[21644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 18:41:53.318601 21725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08: No bootstrap required, opened a new log
I20260619 18:41:53.322664 21725 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1cb3d687914da281d3bb50c12c0c08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38107 } }
I20260619 18:41:53.322891 21725 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:53.322928 21725 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd1cb3d687914da281d3bb50c12c0c08, State: Initialized, Role: FOLLOWER
I20260619 18:41:53.323140 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:53.309552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:51027 (local address 127.0.0.1:38107)
0619 18:41:53.309997 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:53.310017 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:53.310062 (+ 45us) server_negotiation.cc:407] Connection header received
0619 18:41:53.310881 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:53.310920 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:53.311368 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:53.311740 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:53.312412 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:53.313845 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:53.314756 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:53.315439 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:53.315598 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:53.318527 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:53.318532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:53.318545 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:53.318572 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:53.320232 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:53.320826 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:53.320830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:53.320835 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:53.320939 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:53.321224 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:53.321226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:53.321227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:53.321608 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:53.321773 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:53.322093 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 18:41:53.322332 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":102,"threads_started":1}
I20260619 18:41:53.323371 21725 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1cb3d687914da281d3bb50c12c0c08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38107 } }
I20260619 18:41:53.323511 21725 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:53.323575 21725 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:53.323673 21725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:53.324543 21725 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1cb3d687914da281d3bb50c12c0c08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38107 } }
I20260619 18:41:53.324863 21725 leader_election.cc:302] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd1cb3d687914da281d3bb50c12c0c08; no voters:
I20260619 18:41:53.325192 21725 leader_election.cc:288] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:53.325297 21730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:53.325551 21730 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [term 1 LEADER]: Becoming Leader. State: Replica: dd1cb3d687914da281d3bb50c12c0c08, State: Running, Role: LEADER
I20260619 18:41:53.325958 21730 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1cb3d687914da281d3bb50c12c0c08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38107 } }
I20260619 18:41:53.326166 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:41:53.326435 21725 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:53.328606 21731 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd1cb3d687914da281d3bb50c12c0c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1cb3d687914da281d3bb50c12c0c08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38107 } } }
I20260619 18:41:53.328750 21731 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:53.329074 21732 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd1cb3d687914da281d3bb50c12c0c08. Latest consensus state: current_term: 1 leader_uuid: "dd1cb3d687914da281d3bb50c12c0c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1cb3d687914da281d3bb50c12c0c08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38107 } } }
I20260619 18:41:53.329201 21732 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08 [sys.catalog]: This master's current role is: LEADER
W20260619 18:41:53.331920 21748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:41:53.332018 21748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:41:53.332078 21750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:53.332999 21750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:53.337301 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:53.329674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:42963 (local address 127.0.0.1:38107)
0619 18:41:53.329784 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:53.329790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:53.329999 (+ 209us) server_negotiation.cc:407] Connection header received
0619 18:41:53.330132 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:53.330135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:53.330181 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:53.330245 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:53.330907 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:53.331446 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:53.332298 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:53.332534 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:53.332621 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:53.334050 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:53.334054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:53.334057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:53.334083 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:53.335432 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:53.335963 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:53.335967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:53.335969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:53.336063 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:53.336407 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:53.336411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:53.336413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:53.336633 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:53.336792 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:53.337049 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 18:41:53.337151 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 18:41:53.339854 21750 catalog_manager.cc:1347] Generated new cluster ID: 1127f2fe69d34823bd4c2192ce7ca12a
I20260619 18:41:53.339933 21750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:53.357967 21750 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:53.359236 21750 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:53.370586 21750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd1cb3d687914da281d3bb50c12c0c08: Generated new TSK 0
I20260619 18:41:53.371388 21750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:53.387118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 21657
W20260619 18:41:53.391835 21642 connection.cc:535] client connection to 127.0.0.1:38107 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38107): Connection reset by peer (error 104)
2026-06-19T18:41:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:53 dist-test-slave-rxv6 krb5kdc[21756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:53 dist-test-slave-rxv6 krb5kdc[21756](info): set up 2 sockets
Jun 19 18:41:53 dist-test-slave-rxv6 krb5kdc[21756](info): commencing operation
krb5kdc: starting...
W20260619 18:41:54.341890 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:54 dist-test-slave-rxv6 krb5kdc[21756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:41:54.457059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43865
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:54.564465 21769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:54.564733 21769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:54.564775 21769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:54.568850 21769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:54.568924 21769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:54.568943 21769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:54.568959 21769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:54.568974 21769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:54.574440 21769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43865
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:54.575492 21769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:54.576476 21769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:54.584756 21775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:54.584861 21774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:54.584933 21769 server_base.cc:1034] running on GCE node
W20260619 18:41:54.585043 21777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:54.585700 21769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:54.586946 21769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:54.588117 21769 hybrid_clock.cc:648] HybridClock initialized: now 1781894514588090 us; error 49 us; skew 500 ppm
Jun 19 18:41:54 dist-test-slave-rxv6 krb5kdc[21756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:54.591552 21769 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:41:54.592579 21769 webserver.cc:460] Webserver started at http://127.0.0.1:34445/ using document root <none> and password file <none>
I20260619 18:41:54.593083 21769 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:54.593139 21769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:54.593297 21769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:54.595419 21769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "39ec6182393a4ea0bee1081e62f8456b"
format_stamp: "Formatted at 2026-06-19 18:41:54 on dist-test-slave-rxv6"
server_key: "f0082f2af300c4c2bde0c047cfa996c2"
server_key_iv: "7025d8f9a0116c8cb89f21fd6af023d0"
server_key_version: "encryptionkey@0"
I20260619 18:41:54.595948 21769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "39ec6182393a4ea0bee1081e62f8456b"
format_stamp: "Formatted at 2026-06-19 18:41:54 on dist-test-slave-rxv6"
server_key: "f0082f2af300c4c2bde0c047cfa996c2"
server_key_iv: "7025d8f9a0116c8cb89f21fd6af023d0"
server_key_version: "encryptionkey@0"
I20260619 18:41:54.599822 21769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 18:41:54.602433 21784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:54.603720 21769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:41:54.603838 21769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "39ec6182393a4ea0bee1081e62f8456b"
format_stamp: "Formatted at 2026-06-19 18:41:54 on dist-test-slave-rxv6"
server_key: "f0082f2af300c4c2bde0c047cfa996c2"
server_key_iv: "7025d8f9a0116c8cb89f21fd6af023d0"
server_key_version: "encryptionkey@0"
I20260619 18:41:54.603941 21769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:54.629629 21769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:54.632573 21769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:54.632786 21769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:54.639505 21769 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43865
I20260619 18:41:54.639547 21836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43865 every 8 connection(s)
I20260619 18:41:54.640426 21769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:54.643014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 21769
I20260619 18:41:54.643123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:54.643152 21837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:54.643404 392 external_mini_cluster.cc:1428] Setting key da220500d92aeee897caea6de583bce8
I20260619 18:41:54.650094 21837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b: Bootstrap starting.
I20260619 18:41:54.652781 21837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:54.653028 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:54.644733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:60711 (local address 127.0.0.1:43865)
0619 18:41:54.645138 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:54.645156 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:54.645199 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:41:54.645838 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:54.645861 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:54.646320 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:54.646694 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:54.647095 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:54.648513 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:54.649371 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:54.650081 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:54.650229 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:54.651083 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:54.651089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:54.651104 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:54.651598 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:54.651672 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:54.651700 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:54.651822 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:54.651928 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 18:41:54.652073 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":103,"threads_started":1}
I20260619 18:41:54.653589 21837 log.cc:824] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:54.655457 21837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b: No bootstrap required, opened a new log
I20260619 18:41:54.655805 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:41:54.659363 21837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ec6182393a4ea0bee1081e62f8456b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43865 } }
I20260619 18:41:54.659643 21837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:54.659699 21837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39ec6182393a4ea0bee1081e62f8456b, State: Initialized, Role: FOLLOWER
I20260619 18:41:54.660132 21837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ec6182393a4ea0bee1081e62f8456b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43865 } }
I20260619 18:41:54.660269 21837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:54.660328 21837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:54.660418 21837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:54.661474 21837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ec6182393a4ea0bee1081e62f8456b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43865 } }
I20260619 18:41:54.661640 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:54.657619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:36201 (local address 127.0.0.1:43865)
0619 18:41:54.657737 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:54.657741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:54.657893 (+ 152us) server_negotiation.cc:407] Connection header received
0619 18:41:54.658071 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:54.658073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:54.658111 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:54.658174 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:54.658477 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:54.659001 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:54.660034 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:54.660285 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:54.660380 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:54.661250 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:54.661254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:54.661256 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:54.661269 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:54.661298 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:54.661312 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:54.661400 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:54.661490 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 18:41:54.661524 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 18:41:54.661844 21837 leader_election.cc:302] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39ec6182393a4ea0bee1081e62f8456b; no voters:
I20260619 18:41:54.662182 21837 leader_election.cc:288] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:54.662249 21847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:54.662458 21847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [term 1 LEADER]: Becoming Leader. State: Replica: 39ec6182393a4ea0bee1081e62f8456b, State: Running, Role: LEADER
I20260619 18:41:54.662889 21847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ec6182393a4ea0bee1081e62f8456b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43865 } }
I20260619 18:41:54.663367 21837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:54.664009 21847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39ec6182393a4ea0bee1081e62f8456b. Latest consensus state: current_term: 1 leader_uuid: "39ec6182393a4ea0bee1081e62f8456b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ec6182393a4ea0bee1081e62f8456b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43865 } } }
I20260619 18:41:54.664130 21847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:54.664400 21848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39ec6182393a4ea0bee1081e62f8456b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39ec6182393a4ea0bee1081e62f8456b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43865 } } }
I20260619 18:41:54.664500 21848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:54.664592 21856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:54.667454 21856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:54.673678 21856 catalog_manager.cc:1347] Generated new cluster ID: ad7994c2c5954e94a916ac712ebea3ab
I20260619 18:41:54.673751 21856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:54.687331 21856 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:54.688328 21856 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:54.698859 21856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39ec6182393a4ea0bee1081e62f8456b: Generated new TSK 0
I20260619 18:41:54.699525 21856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:54.707230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 21769
W20260619 18:41:54.711843 21754 connection.cc:535] client connection to 127.0.0.1:43865 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43865): Connection reset by peer (error 104)
2026-06-19T18:41:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:54 dist-test-slave-rxv6 krb5kdc[21868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:54 dist-test-slave-rxv6 krb5kdc[21868](info): set up 2 sockets
Jun 19 18:41:54 dist-test-slave-rxv6 krb5kdc[21868](info): commencing operation
krb5kdc: starting...
W20260619 18:41:55.669965 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:55 dist-test-slave-rxv6 krb5kdc[21868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 18:41:55.788820 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41427
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:55.900205 21881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:55.900478 21881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:55.900547 21881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:55.904618 21881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:55.904696 21881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:55.904722 21881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:55.904743 21881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:55.904764 21881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:55.910279 21881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41427
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:55.911438 21881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:55.912552 21881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:55.920696 21887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:55.920780 21886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:55.920822 21889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:55.921074 21881 server_base.cc:1034] running on GCE node
I20260619 18:41:55.921651 21881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:55.922822 21881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:55.923978 21881 hybrid_clock.cc:648] HybridClock initialized: now 1781894515923963 us; error 32 us; skew 500 ppm
Jun 19 18:41:55 dist-test-slave-rxv6 krb5kdc[21868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:55.926771 21881 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 18:41:55.927713 21881 webserver.cc:460] Webserver started at http://127.0.0.1:45075/ using document root <none> and password file <none>
I20260619 18:41:55.928262 21881 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:55.928328 21881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:55.928517 21881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:55.930625 21881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "6361bfcede9146bfa7f1bba81313b75e"
format_stamp: "Formatted at 2026-06-19 18:41:55 on dist-test-slave-rxv6"
server_key: "2dcaa7809fb8596a17ad8f32d68511e3"
server_key_iv: "b064beb8327c525ff28f3e2bc324fa92"
server_key_version: "encryptionkey@0"
I20260619 18:41:55.931186 21881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "6361bfcede9146bfa7f1bba81313b75e"
format_stamp: "Formatted at 2026-06-19 18:41:55 on dist-test-slave-rxv6"
server_key: "2dcaa7809fb8596a17ad8f32d68511e3"
server_key_iv: "b064beb8327c525ff28f3e2bc324fa92"
server_key_version: "encryptionkey@0"
I20260619 18:41:55.934991 21881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:41:55.937541 21896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:55.938915 21881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 18:41:55.939050 21881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "6361bfcede9146bfa7f1bba81313b75e"
format_stamp: "Formatted at 2026-06-19 18:41:55 on dist-test-slave-rxv6"
server_key: "2dcaa7809fb8596a17ad8f32d68511e3"
server_key_iv: "b064beb8327c525ff28f3e2bc324fa92"
server_key_version: "encryptionkey@0"
I20260619 18:41:55.939165 21881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:55.943761 21881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:55.946916 21881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:55.947103 21881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:55.953722 21881 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41427
I20260619 18:41:55.953734 21948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41427 every 8 connection(s)
I20260619 18:41:55.954725 21881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:55.957557 21949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:55.963234 21949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e: Bootstrap starting.
I20260619 18:41:55.964915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 21881
I20260619 18:41:55.965052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:55.965337 392 external_mini_cluster.cc:1428] Setting key 07e08daab59273403d87a518fcaf3bc9
I20260619 18:41:55.965739 21949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:55.966639 21949 log.cc:824] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:55.968537 21949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e: No bootstrap required, opened a new log
I20260619 18:41:55.970312 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:55.966442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:41123 (local address 127.0.0.1:41427)
0619 18:41:55.966955 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:55.966970 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:55.967006 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:41:55.967687 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:55.967705 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:55.967999 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:55.968318 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:55.969084 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:55.969089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:55.969101 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:55.969110 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:55.969164 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:55.969182 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:55.969276 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:55.969395 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 18:41:55.969559 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":152,"threads_started":1}
I20260619 18:41:55.972559 21949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6361bfcede9146bfa7f1bba81313b75e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } }
I20260619 18:41:55.972848 21949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:55.972908 21949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6361bfcede9146bfa7f1bba81313b75e, State: Initialized, Role: FOLLOWER
I20260619 18:41:55.973336 21949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6361bfcede9146bfa7f1bba81313b75e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } }
I20260619 18:41:55.973482 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:41:55.973500 21949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:55.973564 21949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:55.973661 21949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:55.974632 21949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6361bfcede9146bfa7f1bba81313b75e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } }
I20260619 18:41:55.975026 21949 leader_election.cc:302] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6361bfcede9146bfa7f1bba81313b75e; no voters:
I20260619 18:41:55.975350 21949 leader_election.cc:288] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:55.975423 21958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:55.975665 21958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [term 1 LEADER]: Becoming Leader. State: Replica: 6361bfcede9146bfa7f1bba81313b75e, State: Running, Role: LEADER
I20260619 18:41:55.976048 21958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6361bfcede9146bfa7f1bba81313b75e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } }
I20260619 18:41:55.976627 21949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:55.977190 21958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6361bfcede9146bfa7f1bba81313b75e. Latest consensus state: current_term: 1 leader_uuid: "6361bfcede9146bfa7f1bba81313b75e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6361bfcede9146bfa7f1bba81313b75e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } } }
I20260619 18:41:55.977382 21958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:55.977608 21959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6361bfcede9146bfa7f1bba81313b75e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6361bfcede9146bfa7f1bba81313b75e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41427 } } }
I20260619 18:41:55.977728 21959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:55.977851 21967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:55.980504 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:55.975865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:59963 (local address 127.0.0.1:41427)
0619 18:41:55.975991 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:55.975997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:55.978951 (+ 2954us) server_negotiation.cc:407] Connection header received
0619 18:41:55.979213 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:55.979217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:55.979260 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:55.979335 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:55.980062 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:55.980066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:55.980068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:55.980081 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:55.980111 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:55.980124 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:55.980207 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:55.980330 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 18:41:55.980377 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:41:55.981603 21967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:55.987655 21967 catalog_manager.cc:1347] Generated new cluster ID: 962f0028456c494d9fbf78a5044cfbf3
I20260619 18:41:55.987775 21967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:55.999831 21967 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:56.000142 21967 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:56.011623 21967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6361bfcede9146bfa7f1bba81313b75e: Generated new TSK 0
I20260619 18:41:56.012331 21967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:56.017292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 21881
2026-06-19T18:41:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:56 dist-test-slave-rxv6 krb5kdc[21980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:56 dist-test-slave-rxv6 krb5kdc[21980](info): set up 2 sockets
Jun 19 18:41:56 dist-test-slave-rxv6 krb5kdc[21980](info): commencing operation
krb5kdc: starting...
W20260619 18:41:56.969790 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:57 dist-test-slave-rxv6 krb5kdc[21980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:41:57.085498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38831
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:57.202664 21993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:57.202972 21993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:57.203037 21993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:57.207191 21993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:57.207266 21993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:57.207293 21993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:57.207316 21993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:57.207346 21993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:57.213061 21993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38831
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:57.214223 21993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:57.215363 21993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:57.223732 21999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:57.223699 21998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:57.223724 22001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:57.223832 21993 server_base.cc:1034] running on GCE node
I20260619 18:41:57.224568 21993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:57.225656 21993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:57.226833 21993 hybrid_clock.cc:648] HybridClock initialized: now 1781894517226812 us; error 33 us; skew 500 ppm
Jun 19 18:41:57 dist-test-slave-rxv6 krb5kdc[21980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:57.233443 21993 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:41:57.234424 21993 webserver.cc:460] Webserver started at http://127.0.98.62:38993/ using document root <none> and password file <none>
I20260619 18:41:57.234972 21993 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:57.235059 21993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:57.235272 21993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:57.237385 21993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "42bac4bbfab545c0b870f69c95d29522"
format_stamp: "Formatted at 2026-06-19 18:41:57 on dist-test-slave-rxv6"
server_key: "3145873acfc306d29338d43ecb267574"
server_key_iv: "163e2e38b496451ee0bbc8ac7b217cad"
server_key_version: "encryptionkey@0"
I20260619 18:41:57.238170 21993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "42bac4bbfab545c0b870f69c95d29522"
format_stamp: "Formatted at 2026-06-19 18:41:57 on dist-test-slave-rxv6"
server_key: "3145873acfc306d29338d43ecb267574"
server_key_iv: "163e2e38b496451ee0bbc8ac7b217cad"
server_key_version: "encryptionkey@0"
I20260619 18:41:57.242039 21993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 18:41:57.244609 22008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:57.245893 21993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:41:57.246043 21993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "42bac4bbfab545c0b870f69c95d29522"
format_stamp: "Formatted at 2026-06-19 18:41:57 on dist-test-slave-rxv6"
server_key: "3145873acfc306d29338d43ecb267574"
server_key_iv: "163e2e38b496451ee0bbc8ac7b217cad"
server_key_version: "encryptionkey@0"
I20260619 18:41:57.246146 21993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:57.275141 21993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:57.278334 21993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:57.278520 21993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:57.285324 21993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38831
I20260619 18:41:57.285360 22060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38831 every 8 connection(s)
I20260619 18:41:57.286311 21993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:57.289062 22061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:57.292079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 21993
I20260619 18:41:57.292214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:57.292470 392 external_mini_cluster.cc:1428] Setting key 1b6fad10e5e92cf8b912fe14e10c5f5e
I20260619 18:41:57.294902 22061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522: Bootstrap starting.
I20260619 18:41:57.297349 22061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:57.298180 22061 log.cc:824] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:57.300400 22061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522: No bootstrap required, opened a new log
Jun 19 18:41:57 dist-test-slave-rxv6 krb5kdc[21980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:41:57.303964 22061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bac4bbfab545c0b870f69c95d29522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38831 } }
I20260619 18:41:57.304198 22061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:57.304255 22061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42bac4bbfab545c0b870f69c95d29522, State: Initialized, Role: FOLLOWER
I20260619 18:41:57.304697 22061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bac4bbfab545c0b870f69c95d29522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38831 } }
I20260619 18:41:57.304854 22061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:57.304924 22061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:57.305025 22061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:57.306075 22061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bac4bbfab545c0b870f69c95d29522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38831 } }
I20260619 18:41:57.306396 22061 leader_election.cc:302] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42bac4bbfab545c0b870f69c95d29522; no voters:
I20260619 18:41:57.306702 22061 leader_election.cc:288] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:57.306980 22066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:57.307217 22066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [term 1 LEADER]: Becoming Leader. State: Replica: 42bac4bbfab545c0b870f69c95d29522, State: Running, Role: LEADER
I20260619 18:41:57.307559 22066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bac4bbfab545c0b870f69c95d29522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38831 } }
I20260619 18:41:57.307739 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:57.293889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59235 (local address 127.0.98.62:38831)
0619 18:41:57.294330 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:57.294345 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:57.294382 (+ 37us) server_negotiation.cc:407] Connection header received
0619 18:41:57.295010 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:57.295033 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:57.295390 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:57.295671 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:57.296593 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:57.297669 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:57.298855 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:57.299398 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:57.299523 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:57.302445 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:57.302452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:57.302465 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:57.302492 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:57.304632 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:57.305216 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:57.305222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:57.305226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:57.305384 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:57.305698 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:57.305700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:57.305701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:57.306145 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:57.306392 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:57.306620 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 18:41:57.306848 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":104,"threads_started":1}
I20260619 18:41:57.307992 22061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:57.309175 22067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42bac4bbfab545c0b870f69c95d29522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bac4bbfab545c0b870f69c95d29522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38831 } } }
I20260619 18:41:57.309296 22067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:57.309590 22068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42bac4bbfab545c0b870f69c95d29522. Latest consensus state: current_term: 1 leader_uuid: "42bac4bbfab545c0b870f69c95d29522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42bac4bbfab545c0b870f69c95d29522" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38831 } } }
I20260619 18:41:57.309666 22068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522 [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:57.310683 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 18:41:57.312745 22081 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:41:57.312813 22081 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:41:57.312877 22076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:57.313685 22076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:57.319598 22076 catalog_manager.cc:1347] Generated new cluster ID: 755e6bfae0794f01bd4235646a200941
I20260619 18:41:57.319653 22076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:57.324026 22064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:57.312776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36117 (local address 127.0.98.62:38831)
0619 18:41:57.312890 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:57.312894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:57.313162 (+ 268us) server_negotiation.cc:407] Connection header received
0619 18:41:57.313276 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:57.313279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:57.313327 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:57.313408 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:57.314188 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:57.314940 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:57.316081 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:57.316396 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:57.316498 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:57.318098 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:57.318101 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:57.318104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:41:57.318128 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:57.322314 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:57.322833 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:57.322837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:57.322839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:57.322925 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:41:57.323207 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:41:57.323210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:41:57.323212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:41:57.323425 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:57.323573 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:57.323777 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 18:41:57.323872 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:41:57.341012 22076 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:57.342077 22076 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:57.348863 22076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42bac4bbfab545c0b870f69c95d29522: Generated new TSK 0
I20260619 18:41:57.349550 22076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:57.361768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 21993
W20260619 18:41:57.366677 21978 connection.cc:535] client connection to 127.0.98.62:38831 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:38831): Connection reset by peer (error 104)
2026-06-19T18:41:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:57 dist-test-slave-rxv6 krb5kdc[22092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:57 dist-test-slave-rxv6 krb5kdc[22092](info): set up 2 sockets
Jun 19 18:41:57 dist-test-slave-rxv6 krb5kdc[22092](info): commencing operation
krb5kdc: starting...
W20260619 18:41:58.317885 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:58 dist-test-slave-rxv6 krb5kdc[22092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:41:58.428826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45435
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:58.534763 22105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:58.535039 22105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:58.535097 22105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:58.539608 22105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:58.539675 22105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:58.539702 22105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:58.539723 22105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:58.539745 22105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:58.545029 22105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45435
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:58.546082 22105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:58.547034 22105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:58.554975 22111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:58.555089 22105 server_base.cc:1034] running on GCE node
W20260619 18:41:58.554965 22110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:58.555302 22113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:58.555822 22105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:58.556893 22105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:58.558048 22105 hybrid_clock.cc:648] HybridClock initialized: now 1781894518558023 us; error 39 us; skew 500 ppm
Jun 19 18:41:58 dist-test-slave-rxv6 krb5kdc[22092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:58.560668 22105 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:41:58.561501 22105 webserver.cc:460] Webserver started at http://127.0.98.62:44495/ using document root <none> and password file <none>
I20260619 18:41:58.562054 22105 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:58.562119 22105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:58.562287 22105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:58.564307 22105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "909660fc20dd49b2b2dd208c08beacca"
format_stamp: "Formatted at 2026-06-19 18:41:58 on dist-test-slave-rxv6"
server_key: "9b8ea5fd4c3d8c7a7ec4f819e924af97"
server_key_iv: "7c5aac68e4793107a3f0053ace3b679f"
server_key_version: "encryptionkey@0"
I20260619 18:41:58.564796 22105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "909660fc20dd49b2b2dd208c08beacca"
format_stamp: "Formatted at 2026-06-19 18:41:58 on dist-test-slave-rxv6"
server_key: "9b8ea5fd4c3d8c7a7ec4f819e924af97"
server_key_iv: "7c5aac68e4793107a3f0053ace3b679f"
server_key_version: "encryptionkey@0"
I20260619 18:41:58.568464 22105 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:41:58.570813 22120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:58.571959 22105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 18:41:58.572054 22105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "909660fc20dd49b2b2dd208c08beacca"
format_stamp: "Formatted at 2026-06-19 18:41:58 on dist-test-slave-rxv6"
server_key: "9b8ea5fd4c3d8c7a7ec4f819e924af97"
server_key_iv: "7c5aac68e4793107a3f0053ace3b679f"
server_key_version: "encryptionkey@0"
I20260619 18:41:58.572139 22105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:58.591732 22105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:58.594794 22105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:58.594940 22105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:58.601215 22105 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45435
I20260619 18:41:58.601264 22172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45435 every 8 connection(s)
I20260619 18:41:58.602123 22105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:58.604451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 22105
I20260619 18:41:58.604568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:58.604832 22173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:58.604846 392 external_mini_cluster.cc:1428] Setting key b1a48fd76617a65054eed233c30e85bd
I20260619 18:41:58.611896 22173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca: Bootstrap starting.
I20260619 18:41:58.614387 22173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:58.614852 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:58.606182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58081 (local address 127.0.98.62:45435)
0619 18:41:58.606602 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:58.606620 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:58.606662 (+ 42us) server_negotiation.cc:407] Connection header received
0619 18:41:58.607502 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:58.607526 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:58.607961 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:58.608342 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:58.608749 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:58.610241 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:58.611111 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:58.611795 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:58.611956 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:58.612859 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:58.612865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:58.612881 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:58.613372 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:58.613434 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:58.613463 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:58.613590 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:58.613777 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 18:41:58.613937 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":122,"threads_started":1}
I20260619 18:41:58.615204 22173 log.cc:824] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:58.617089 22173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca: No bootstrap required, opened a new log
I20260619 18:41:58.617515 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:41:58.620589 22173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909660fc20dd49b2b2dd208c08beacca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } }
I20260619 18:41:58.620793 22173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:58.620853 22173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909660fc20dd49b2b2dd208c08beacca, State: Initialized, Role: FOLLOWER
I20260619 18:41:58.621240 22173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909660fc20dd49b2b2dd208c08beacca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } }
I20260619 18:41:58.621369 22173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:58.621433 22173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:58.621521 22173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:58.622512 22173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909660fc20dd49b2b2dd208c08beacca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } }
I20260619 18:41:58.622855 22173 leader_election.cc:302] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 909660fc20dd49b2b2dd208c08beacca; no voters:
I20260619 18:41:58.623003 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:58.619340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37269 (local address 127.0.98.62:45435)
0619 18:41:58.619453 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:58.619457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:58.619641 (+ 184us) server_negotiation.cc:407] Connection header received
0619 18:41:58.619742 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:58.619745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:58.619784 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:58.619844 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:58.620072 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:58.620559 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:58.621403 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:41:58.621731 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:41:58.621837 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:41:58.622581 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:58.622585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:58.622588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:58.622600 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:58.622625 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:58.622637 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:58.622718 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:58.622855 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 18:41:58.622889 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:41:58.623169 22173 leader_election.cc:288] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:58.623260 22183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:58.623490 22183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [term 1 LEADER]: Becoming Leader. State: Replica: 909660fc20dd49b2b2dd208c08beacca, State: Running, Role: LEADER
I20260619 18:41:58.623853 22183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909660fc20dd49b2b2dd208c08beacca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } }
I20260619 18:41:58.624315 22173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:58.624939 22183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 909660fc20dd49b2b2dd208c08beacca. Latest consensus state: current_term: 1 leader_uuid: "909660fc20dd49b2b2dd208c08beacca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909660fc20dd49b2b2dd208c08beacca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } } }
I20260619 18:41:58.625047 22183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:58.625636 22184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "909660fc20dd49b2b2dd208c08beacca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "909660fc20dd49b2b2dd208c08beacca" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45435 } } }
I20260619 18:41:58.625723 22184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:58.626025 22189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:58.628566 22189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:58.633524 22189 catalog_manager.cc:1347] Generated new cluster ID: bf7744b106df4548979c7f2ee7af7874
I20260619 18:41:58.633605 22189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:58.649127 22189 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:58.650127 22189 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:58.657842 22189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 909660fc20dd49b2b2dd208c08beacca: Generated new TSK 0
I20260619 18:41:58.658519 22189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:58.669548 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 22105
W20260619 18:41:58.674203 22090 connection.cc:535] client connection to 127.0.98.62:45435 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:45435): Connection reset by peer (error 104)
2026-06-19T18:41:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:41:58 dist-test-slave-rxv6 krb5kdc[22204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:41:58 dist-test-slave-rxv6 krb5kdc[22204](info): set up 2 sockets
Jun 19 18:41:58 dist-test-slave-rxv6 krb5kdc[22204](info): commencing operation
krb5kdc: starting...
W20260619 18:41:59.621932 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:41:59 dist-test-slave-rxv6 krb5kdc[22204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:41:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:41:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:41:59.734720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39533
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:41:59.844619 22217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:41:59.844890 22217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:41:59.844951 22217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:41:59.849016 22217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:41:59.849089 22217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:41:59.849117 22217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:41:59.849139 22217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:41:59.849164 22217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:41:59.854456 22217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:41:59.855470 22217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:41:59.856436 22217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:41:59.864689 22223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:41:59.864714 22222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:59.864797 22217 server_base.cc:1034] running on GCE node
W20260619 18:41:59.864696 22225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:41:59.865546 22217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:41:59.866734 22217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:41:59.867911 22217 hybrid_clock.cc:648] HybridClock initialized: now 1781894519867890 us; error 34 us; skew 500 ppm
Jun 19 18:41:59 dist-test-slave-rxv6 krb5kdc[22204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:41:59.870699 22217 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:41:59.871565 22217 webserver.cc:460] Webserver started at http://127.0.98.62:45441/ using document root <none> and password file <none>
I20260619 18:41:59.872098 22217 fs_manager.cc:359] Metadata directory not provided
I20260619 18:41:59.872170 22217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:41:59.872385 22217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:41:59.874605 22217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "562d37f2f851408a93c8708472d2770a"
format_stamp: "Formatted at 2026-06-19 18:41:59 on dist-test-slave-rxv6"
server_key: "5736b45b1473ece4edc7c955cca05624"
server_key_iv: "d7b67752d522d080a82b5e28e0832b2d"
server_key_version: "encryptionkey@0"
I20260619 18:41:59.875180 22217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "562d37f2f851408a93c8708472d2770a"
format_stamp: "Formatted at 2026-06-19 18:41:59 on dist-test-slave-rxv6"
server_key: "5736b45b1473ece4edc7c955cca05624"
server_key_iv: "d7b67752d522d080a82b5e28e0832b2d"
server_key_version: "encryptionkey@0"
I20260619 18:41:59.879191 22217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 18:41:59.882051 22232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:41:59.883379 22217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:41:59.883548 22217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "562d37f2f851408a93c8708472d2770a"
format_stamp: "Formatted at 2026-06-19 18:41:59 on dist-test-slave-rxv6"
server_key: "5736b45b1473ece4edc7c955cca05624"
server_key_iv: "d7b67752d522d080a82b5e28e0832b2d"
server_key_version: "encryptionkey@0"
I20260619 18:41:59.883710 22217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:41:59.888437 22217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:41:59.891335 22217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:41:59.891510 22217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:41:59.898131 22217 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39533
I20260619 18:41:59.898162 22284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39533 every 8 connection(s)
I20260619 18:41:59.899066 22217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:41:59.900512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 22217
I20260619 18:41:59.900656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:41:59.900936 392 external_mini_cluster.cc:1428] Setting key 7d1c9e713e59c6cec7ede37fe68a7c0e
I20260619 18:41:59.901960 22285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:41:59.906446 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:59.902313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52581 (local address 127.0.98.62:39533)
0619 18:41:59.902716 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:59.902734 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:59.902777 (+ 43us) server_negotiation.cc:407] Connection header received
0619 18:41:59.903593 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:59.903618 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:59.903964 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:59.904346 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:59.905042 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:59.905048 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:59.905062 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:59.905074 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:59.905131 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:59.905157 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:59.905304 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:59.905370 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 18:41:59.905522 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
I20260619 18:41:59.908579 22285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a: Bootstrap starting.
I20260619 18:41:59.909332 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:41:59.911229 22285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a: Neither blocks nor log segments found. Creating new log.
I20260619 18:41:59.912107 22285 log.cc:824] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a: Log is configured to *not* fsync() on all Append() calls
I20260619 18:41:59.913117 22288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:41:59.911518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47637 (local address 127.0.98.62:39533)
0619 18:41:59.911638 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 18:41:59.911643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:41:59.911864 (+ 221us) server_negotiation.cc:407] Connection header received
0619 18:41:59.911978 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:41:59.911980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:41:59.912011 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:41:59.912101 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:41:59.912746 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:41:59.912748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:41:59.912750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:41:59.912760 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:41:59.912783 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:41:59.912793 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:41:59.912868 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 18:41:59.912943 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 18:41:59.912980 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 18:41:59.914326 22285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a: No bootstrap required, opened a new log
I20260619 18:41:59.917727 22285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562d37f2f851408a93c8708472d2770a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39533 } }
I20260619 18:41:59.917995 22285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:41:59.918054 22285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562d37f2f851408a93c8708472d2770a, State: Initialized, Role: FOLLOWER
I20260619 18:41:59.918454 22285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562d37f2f851408a93c8708472d2770a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39533 } }
I20260619 18:41:59.918584 22285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:41:59.918648 22285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:41:59.918743 22285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:41:59.919612 22285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562d37f2f851408a93c8708472d2770a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39533 } }
I20260619 18:41:59.919924 22285 leader_election.cc:302] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 562d37f2f851408a93c8708472d2770a; no voters:
I20260619 18:41:59.920234 22285 leader_election.cc:288] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:41:59.920331 22295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:41:59.920524 22295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [term 1 LEADER]: Becoming Leader. State: Replica: 562d37f2f851408a93c8708472d2770a, State: Running, Role: LEADER
I20260619 18:41:59.920883 22295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562d37f2f851408a93c8708472d2770a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39533 } }
I20260619 18:41:59.921345 22285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:41:59.922381 22296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "562d37f2f851408a93c8708472d2770a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562d37f2f851408a93c8708472d2770a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39533 } } }
I20260619 18:41:59.922508 22296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:59.922914 22304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:41:59.922883 22297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 562d37f2f851408a93c8708472d2770a. Latest consensus state: current_term: 1 leader_uuid: "562d37f2f851408a93c8708472d2770a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "562d37f2f851408a93c8708472d2770a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39533 } } }
I20260619 18:41:59.923069 22297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a [sys.catalog]: This master's current role is: LEADER
I20260619 18:41:59.925721 22304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:41:59.931496 22304 catalog_manager.cc:1347] Generated new cluster ID: 1e6d75f13e554a99b94cde09c83160e1
I20260619 18:41:59.931646 22304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:41:59.948665 22304 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:41:59.949015 22304 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:41:59.954784 22304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 562d37f2f851408a93c8708472d2770a: Generated new TSK 0
I20260619 18:41:59.955492 22304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:41:59.959290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 22217
2026-06-19T18:41:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1290 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:00 dist-test-slave-rxv6 krb5kdc[22316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:00 dist-test-slave-rxv6 krb5kdc[22316](info): set up 2 sockets
Jun 19 18:42:00 dist-test-slave-rxv6 krb5kdc[22316](info): commencing operation
krb5kdc: starting...
W20260619 18:42:00.917899 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:42:00 dist-test-slave-rxv6 krb5kdc[22316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:42:01.030972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40497
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:01.140106 22329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:01.140398 22329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:01.140456 22329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:01.144572 22329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:01.144650 22329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:01.144677 22329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:01.144698 22329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:01.144734 22329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:01.150156 22329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:01.151324 22329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:01.152336 22329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:01.160462 22334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:01.160460 22337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:01.160562 22335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:01.160725 22329 server_base.cc:1034] running on GCE node
I20260619 18:42:01.161355 22329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:01.162484 22329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:01.163654 22329 hybrid_clock.cc:648] HybridClock initialized: now 1781894521163638 us; error 29 us; skew 500 ppm
Jun 19 18:42:01 dist-test-slave-rxv6 krb5kdc[22316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:01.166455 22329 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:42:01.167407 22329 webserver.cc:460] Webserver started at http://127.0.98.62:39727/ using document root <none> and password file <none>
I20260619 18:42:01.167941 22329 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:01.168052 22329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:01.168263 22329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:01.170344 22329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "484e2bba1eef457a9a246145c53cacc2"
format_stamp: "Formatted at 2026-06-19 18:42:01 on dist-test-slave-rxv6"
server_key: "4afe996148ea35ac34b5b74b51a5605b"
server_key_iv: "e9222af27b5fbc9d4cd3d0f11e1812ec"
server_key_version: "encryptionkey@0"
I20260619 18:42:01.170908 22329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "484e2bba1eef457a9a246145c53cacc2"
format_stamp: "Formatted at 2026-06-19 18:42:01 on dist-test-slave-rxv6"
server_key: "4afe996148ea35ac34b5b74b51a5605b"
server_key_iv: "e9222af27b5fbc9d4cd3d0f11e1812ec"
server_key_version: "encryptionkey@0"
I20260619 18:42:01.174595 22329 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:42:01.176968 22344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:01.178251 22329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:42:01.178378 22329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "484e2bba1eef457a9a246145c53cacc2"
format_stamp: "Formatted at 2026-06-19 18:42:01 on dist-test-slave-rxv6"
server_key: "4afe996148ea35ac34b5b74b51a5605b"
server_key_iv: "e9222af27b5fbc9d4cd3d0f11e1812ec"
server_key_version: "encryptionkey@0"
I20260619 18:42:01.178483 22329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:01.197449 22329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:01.200165 22329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:01.200341 22329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:01.206938 22329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40497
I20260619 18:42:01.206967 22396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40497 every 8 connection(s)
I20260619 18:42:01.207916 22329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:01.210673 22397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:01.216068 22397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2: Bootstrap starting.
I20260619 18:42:01.216701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 22329
I20260619 18:42:01.216820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:42:01.217108 392 external_mini_cluster.cc:1428] Setting key 60d4b34b62c01f861e9f9d617b8f4a71
I20260619 18:42:01.218756 22397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:01.219435 22397 log.cc:824] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:01.221261 22397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2: No bootstrap required, opened a new log
I20260619 18:42:01.224664 22397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484e2bba1eef457a9a246145c53cacc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } }
I20260619 18:42:01.224948 22397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:01.225015 22397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484e2bba1eef457a9a246145c53cacc2, State: Initialized, Role: FOLLOWER
Jun 19 18:42:01 dist-test-slave-rxv6 krb5kdc[22316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:42:01.225498 22397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484e2bba1eef457a9a246145c53cacc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } }
I20260619 18:42:01.225632 22397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:01.225696 22397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:01.225785 22397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:01.226675 22397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484e2bba1eef457a9a246145c53cacc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } }
I20260619 18:42:01.226997 22397 leader_election.cc:302] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 484e2bba1eef457a9a246145c53cacc2; no voters:
I20260619 18:42:01.227289 22397 leader_election.cc:288] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:01.227370 22402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:01.227533 22402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [term 1 LEADER]: Becoming Leader. State: Replica: 484e2bba1eef457a9a246145c53cacc2, State: Running, Role: LEADER
I20260619 18:42:01.227881 22402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484e2bba1eef457a9a246145c53cacc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } }
I20260619 18:42:01.228600 22397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:01.229481 22403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "484e2bba1eef457a9a246145c53cacc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484e2bba1eef457a9a246145c53cacc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } } }
I20260619 18:42:01.229856 22403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:01.229743 22404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 484e2bba1eef457a9a246145c53cacc2. Latest consensus state: current_term: 1 leader_uuid: "484e2bba1eef457a9a246145c53cacc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "484e2bba1eef457a9a246145c53cacc2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } } }
I20260619 18:42:01.230163 22404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:01.231009 22411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:01.231863 22400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:01.218249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37069 (local address 127.0.98.62:40497)
0619 18:42:01.218658 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:01.218674 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:01.218708 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:42:01.219363 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:01.219381 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:01.219754 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:01.220065 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:01.220832 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:01.221881 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:01.222743 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:01.223230 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:01.223374 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:01.225896 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:01.225926 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:01.225944 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:01.225978 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:01.227841 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:01.228291 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:01.228297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:01.228303 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:01.228420 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:01.228740 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:01.228744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:01.228745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:01.229242 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:01.230202 (+ 960us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:01.230358 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 18:42:01.230688 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
I20260619 18:42:01.234306 22411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:01.234566 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:42:01.239986 22411 catalog_manager.cc:1347] Generated new cluster ID: 5171df1eedd6472aa225b19f8f02b53d
I20260619 18:42:01.240051 22411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:01.245095 22400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:01.236355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41201 (local address 127.0.98.62:40497)
0619 18:42:01.236475 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:01.236480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:01.236834 (+ 354us) server_negotiation.cc:407] Connection header received
0619 18:42:01.236958 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:01.236961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:01.237009 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:01.237087 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:01.237925 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:01.238619 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:01.239796 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:01.240165 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:01.240243 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:01.241975 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:01.241979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:01.241981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:01.242006 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:01.243536 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:01.243978 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:01.243981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:01.243983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:01.244052 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:01.244299 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:01.244302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:01.244303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:01.244487 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:01.244595 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:01.244821 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 18:42:01.244938 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:42:01.247187 22411 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:01.247989 22411 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:01.252929 22411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 484e2bba1eef457a9a246145c53cacc2: Generated new TSK 0
I20260619 18:42:01.253576 22411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:42:01.259771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 22329
2026-06-19T18:42:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:01 dist-test-slave-rxv6 krb5kdc[22428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:01 dist-test-slave-rxv6 krb5kdc[22428](info): set up 2 sockets
Jun 19 18:42:01 dist-test-slave-rxv6 krb5kdc[22428](info): commencing operation
krb5kdc: starting...
W20260619 18:42:02.210247 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:42:02 dist-test-slave-rxv6 krb5kdc[22428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:42:02.323303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45125
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:02.430917 22441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:02.431174 22441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:02.431216 22441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:02.435251 22441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:02.435315 22441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:02.435333 22441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:02.435353 22441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:02.435380 22441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:02.440563 22441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:02.441568 22441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:02.442570 22441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:02.450773 22446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:02.450773 22447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:02.450805 22449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:02.451014 22441 server_base.cc:1034] running on GCE node
I20260619 18:42:02.451545 22441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:02.452710 22441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:02.453962 22441 hybrid_clock.cc:648] HybridClock initialized: now 1781894522453936 us; error 36 us; skew 500 ppm
Jun 19 18:42:02 dist-test-slave-rxv6 krb5kdc[22428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:02.459756 22441 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:42:02.460654 22441 webserver.cc:460] Webserver started at http://127.0.98.62:39461/ using document root <none> and password file <none>
I20260619 18:42:02.461187 22441 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:02.461251 22441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:02.461450 22441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:02.463517 22441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "6ad364d646e94fecb160e7f406fa0484"
format_stamp: "Formatted at 2026-06-19 18:42:02 on dist-test-slave-rxv6"
server_key: "56cdda6224601744c512dd2f11be7ca6"
server_key_iv: "233d9bb05d58025a2b7dbeda8b86f05e"
server_key_version: "encryptionkey@0"
I20260619 18:42:02.464040 22441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "6ad364d646e94fecb160e7f406fa0484"
format_stamp: "Formatted at 2026-06-19 18:42:02 on dist-test-slave-rxv6"
server_key: "56cdda6224601744c512dd2f11be7ca6"
server_key_iv: "233d9bb05d58025a2b7dbeda8b86f05e"
server_key_version: "encryptionkey@0"
I20260619 18:42:02.467566 22441 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 18:42:02.469841 22456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:02.471058 22441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:02.471174 22441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "6ad364d646e94fecb160e7f406fa0484"
format_stamp: "Formatted at 2026-06-19 18:42:02 on dist-test-slave-rxv6"
server_key: "56cdda6224601744c512dd2f11be7ca6"
server_key_iv: "233d9bb05d58025a2b7dbeda8b86f05e"
server_key_version: "encryptionkey@0"
I20260619 18:42:02.471268 22441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:02.491379 22441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:02.494150 22441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:02.494334 22441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:02.500959 22441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45125
I20260619 18:42:02.500988 22508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45125 every 8 connection(s)
I20260619 18:42:02.501870 22441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:02.504524 22509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:02.508842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 22441
I20260619 18:42:02.508978 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:42:02.509279 392 external_mini_cluster.cc:1428] Setting key 7ce7f0480e4a3d6eef38f7053b94568c
I20260619 18:42:02.510076 22509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484: Bootstrap starting.
I20260619 18:42:02.512660 22509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:02.513368 22509 log.cc:824] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:02.515092 22509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484: No bootstrap required, opened a new log
I20260619 18:42:02.518046 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:02.510693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38479 (local address 127.0.98.62:45125)
0619 18:42:02.511108 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:02.511122 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:02.511155 (+ 33us) server_negotiation.cc:407] Connection header received
0619 18:42:02.511794 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:02.511813 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:02.512255 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:02.512617 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:02.513080 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:02.514113 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:02.514992 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:02.515479 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:02.515622 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:02.516469 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:02.516474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:02.516485 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:02.516852 (+ 367us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:02.516903 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:02.516925 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:02.517007 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:02.517149 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 18:42:02.517340 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":122,"threads_started":1}
I20260619 18:42:02.518684 22509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad364d646e94fecb160e7f406fa0484" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45125 } }
I20260619 18:42:02.518900 22509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:02.519003 22509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad364d646e94fecb160e7f406fa0484, State: Initialized, Role: FOLLOWER
I20260619 18:42:02.519444 22509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad364d646e94fecb160e7f406fa0484" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45125 } }
I20260619 18:42:02.519590 22509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:02.519656 22509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:02.519788 22509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:02.520597 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:42:02.520751 22509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad364d646e94fecb160e7f406fa0484" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45125 } }
I20260619 18:42:02.521088 22509 leader_election.cc:302] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ad364d646e94fecb160e7f406fa0484; no voters:
I20260619 18:42:02.521389 22509 leader_election.cc:288] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:02.521476 22514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:02.521674 22514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [term 1 LEADER]: Becoming Leader. State: Replica: 6ad364d646e94fecb160e7f406fa0484, State: Running, Role: LEADER
I20260619 18:42:02.522028 22514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad364d646e94fecb160e7f406fa0484" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45125 } }
I20260619 18:42:02.522634 22509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:02.523288 22516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ad364d646e94fecb160e7f406fa0484. Latest consensus state: current_term: 1 leader_uuid: "6ad364d646e94fecb160e7f406fa0484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad364d646e94fecb160e7f406fa0484" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45125 } } }
I20260619 18:42:02.523420 22516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:02.523712 22515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ad364d646e94fecb160e7f406fa0484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ad364d646e94fecb160e7f406fa0484" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45125 } } }
I20260619 18:42:02.523811 22515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:02.525002 22526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:02.527346 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:02.522633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46693 (local address 127.0.98.62:45125)
0619 18:42:02.522744 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:02.522748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:02.523176 (+ 428us) server_negotiation.cc:407] Connection header received
0619 18:42:02.523809 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:02.523812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:02.523857 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:02.523906 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:02.524288 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:02.524992 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:02.525883 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:02.526149 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:02.526235 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:02.526961 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:02.526965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:02.526968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:02.526980 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:02.527006 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:02.527016 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:02.527074 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:02.527158 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 18:42:02.527205 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 18:42:02.528133 22526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:02.534372 22526 catalog_manager.cc:1347] Generated new cluster ID: c0bb0964b26c42628b8bdd5fe621495c
I20260619 18:42:02.534444 22526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:02.575546 22526 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:02.576798 22526 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:02.592074 22526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ad364d646e94fecb160e7f406fa0484: Generated new TSK 0
I20260619 18:42:02.592971 22526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:42:02.609145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 22441
2026-06-19T18:42:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:02 dist-test-slave-rxv6 krb5kdc[22540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:02 dist-test-slave-rxv6 krb5kdc[22540](info): set up 2 sockets
Jun 19 18:42:02 dist-test-slave-rxv6 krb5kdc[22540](info): commencing operation
krb5kdc: starting...
W20260619 18:42:03.565889 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:42:03 dist-test-slave-rxv6 krb5kdc[22540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:42:03.682993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37151
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:03.790787 22553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:03.791071 22553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:03.791112 22553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:03.796489 22553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:03.796574 22553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:03.796600 22553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:03.796623 22553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:03.796710 22553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:03.803598 22553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:03.804659 22553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:03.805650 22553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:03.813778 22558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:03.813869 22553 server_base.cc:1034] running on GCE node
W20260619 18:42:03.813829 22561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:03.813786 22559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:03.814568 22553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:03.815629 22553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:03.816789 22553 hybrid_clock.cc:648] HybridClock initialized: now 1781894523816773 us; error 33 us; skew 500 ppm
Jun 19 18:42:03 dist-test-slave-rxv6 krb5kdc[22540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894523, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:03.819368 22553 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:42:03.820151 22553 webserver.cc:460] Webserver started at http://127.0.98.62:40725/ using document root <none> and password file <none>
I20260619 18:42:03.820647 22553 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:03.820703 22553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:03.820900 22553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:03.823196 22553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "ccd98a0470e64a21b31a53655cfd1ec1"
format_stamp: "Formatted at 2026-06-19 18:42:03 on dist-test-slave-rxv6"
server_key: "96c8fd11aae7b44515c1e40eca7678d9"
server_key_iv: "9add21a2ab7651724847d40f65ce7bbe"
server_key_version: "encryptionkey@0"
I20260619 18:42:03.823733 22553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "ccd98a0470e64a21b31a53655cfd1ec1"
format_stamp: "Formatted at 2026-06-19 18:42:03 on dist-test-slave-rxv6"
server_key: "96c8fd11aae7b44515c1e40eca7678d9"
server_key_iv: "9add21a2ab7651724847d40f65ce7bbe"
server_key_version: "encryptionkey@0"
I20260619 18:42:03.827473 22553 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 18:42:03.830163 22568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:03.831534 22553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:42:03.831660 22553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "ccd98a0470e64a21b31a53655cfd1ec1"
format_stamp: "Formatted at 2026-06-19 18:42:03 on dist-test-slave-rxv6"
server_key: "96c8fd11aae7b44515c1e40eca7678d9"
server_key_iv: "9add21a2ab7651724847d40f65ce7bbe"
server_key_version: "encryptionkey@0"
I20260619 18:42:03.831836 22553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:03.836392 22553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:03.839352 22553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:03.839497 22553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:03.846058 22553 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37151
I20260619 18:42:03.846087 22620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37151 every 8 connection(s)
I20260619 18:42:03.846947 22553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:03.848868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 22553
I20260619 18:42:03.848999 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:42:03.849287 392 external_mini_cluster.cc:1428] Setting key bce2d73b80cd9e6f3febce24e05c52f3
I20260619 18:42:03.849848 22621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:03.854333 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:03.850637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53153 (local address 127.0.98.62:37151)
0619 18:42:03.851058 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:03.851073 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:03.851107 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:42:03.851771 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:03.851787 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:03.852104 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:03.852422 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:03.853170 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:03.853181 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:03.853192 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:03.853203 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:03.853255 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:03.853273 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:03.853394 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:03.853466 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 18:42:03.853621 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":141,"threads_started":1}
I20260619 18:42:03.855871 22621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1: Bootstrap starting.
I20260619 18:42:03.856843 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:42:03.858395 22621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:03.859328 22621 log.cc:824] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:03.860625 22624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:03.859043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43171 (local address 127.0.98.62:37151)
0619 18:42:03.859152 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:03.859155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:03.859309 (+ 154us) server_negotiation.cc:407] Connection header received
0619 18:42:03.859458 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:03.859462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:03.859498 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:03.859561 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:03.860288 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:03.860291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:03.860293 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 18:42:03.860302 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:03.860326 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 18:42:03.860337 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:03.860404 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:03.860494 (+ 90us) server_negotiation.cc:299] Negotiation successful
0619 18:42:03.860522 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 18:42:03.861534 22621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1: No bootstrap required, opened a new log
I20260619 18:42:03.865226 22621 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd98a0470e64a21b31a53655cfd1ec1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37151 } }
I20260619 18:42:03.865471 22621 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 18:42:03.865522 22621 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccd98a0470e64a21b31a53655cfd1ec1, State: Initialized, Role: FOLLOWER
I20260619 18:42:03.865947 22621 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd98a0470e64a21b31a53655cfd1ec1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37151 } }
I20260619 18:42:03.866071 22621 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 18:42:03.866132 22621 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 18:42:03.866231 22621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:03.867233 22621 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd98a0470e64a21b31a53655cfd1ec1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37151 } }
I20260619 18:42:03.867555 22621 leader_election.cc:302] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccd98a0470e64a21b31a53655cfd1ec1; no voters:
I20260619 18:42:03.867970 22621 leader_election.cc:288] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 18:42:03.868062 22631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:03.868248 22631 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [term 1 LEADER]: Becoming Leader. State: Replica: ccd98a0470e64a21b31a53655cfd1ec1, State: Running, Role: LEADER
I20260619 18:42:03.868571 22631 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd98a0470e64a21b31a53655cfd1ec1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37151 } }
I20260619 18:42:03.869134 22621 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:03.870239 22633 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccd98a0470e64a21b31a53655cfd1ec1. Latest consensus state: current_term: 1 leader_uuid: "ccd98a0470e64a21b31a53655cfd1ec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd98a0470e64a21b31a53655cfd1ec1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37151 } } }
I20260619 18:42:03.870379 22633 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:03.870216 22632 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccd98a0470e64a21b31a53655cfd1ec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccd98a0470e64a21b31a53655cfd1ec1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37151 } } }
I20260619 18:42:03.870620 22632 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:03.870872 22640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:03.873821 22640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:03.879487 22640 catalog_manager.cc:1347] Generated new cluster ID: 8a3e255fa838484d9a600a28268e2f8c
I20260619 18:42:03.879544 22640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:03.905558 22640 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:03.905858 22640 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:03.910343 22640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccd98a0470e64a21b31a53655cfd1ec1: Generated new TSK 0
I20260619 18:42:03.911095 22640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 18:42:03.924286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 22553
2026-06-19T18:42:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1315 ms)
[----------] 18 tests from AuthTokenIssuingTest (23690 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:03 dist-test-slave-rxv6 krb5kdc[22652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:03 dist-test-slave-rxv6 krb5kdc[22652](info): set up 2 sockets
Jun 19 18:42:03 dist-test-slave-rxv6 krb5kdc[22652](info): commencing operation
krb5kdc: starting...
W20260619 18:42:04.878054 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:42:04 dist-test-slave-rxv6 krb5kdc[22652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:42:04.988533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46421
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46421,127.0.98.61:35777,127.0.98.60:37671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:05.095098 22665 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:42:05.095378 22665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:05.095412 22665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:05.095469 22665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:05.099393 22665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:05.099452 22665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:05.099478 22665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:05.099498 22665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:05.099517 22665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:05.104627 22665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46421,127.0.98.61:35777,127.0.98.60:37671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:05.105656 22665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:05.106668 22665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:05.114885 22670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:05.114918 22673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:05.114913 22671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:05.114959 22665 server_base.cc:1034] running on GCE node
I20260619 18:42:05.115653 22665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:05.116744 22665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:05.117934 22665 hybrid_clock.cc:648] HybridClock initialized: now 1781894525117884 us; error 41 us; skew 500 ppm
Jun 19 18:42:05 dist-test-slave-rxv6 krb5kdc[22652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:05.120503 22665 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:42:05.121387 22665 webserver.cc:460] Webserver started at http://127.0.98.62:46455/ using document root <none> and password file <none>
I20260619 18:42:05.121897 22665 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:05.122008 22665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:05.122255 22665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:05.124262 22665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "b8b1cca3ad0948429fdc02a995a40433"
format_stamp: "Formatted at 2026-06-19 18:42:05 on dist-test-slave-rxv6"
server_key: "1d4f1bf74731eadf6f9cb7b3ca0b3fbf"
server_key_iv: "c91c35f7da2138650834c97e612a3194"
server_key_version: "encryptionkey@0"
I20260619 18:42:05.124785 22665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "b8b1cca3ad0948429fdc02a995a40433"
format_stamp: "Formatted at 2026-06-19 18:42:05 on dist-test-slave-rxv6"
server_key: "1d4f1bf74731eadf6f9cb7b3ca0b3fbf"
server_key_iv: "c91c35f7da2138650834c97e612a3194"
server_key_version: "encryptionkey@0"
I20260619 18:42:05.128479 22665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:42:05.130846 22680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:05.132112 22665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 18:42:05.132236 22665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "b8b1cca3ad0948429fdc02a995a40433"
format_stamp: "Formatted at 2026-06-19 18:42:05 on dist-test-slave-rxv6"
server_key: "1d4f1bf74731eadf6f9cb7b3ca0b3fbf"
server_key_iv: "c91c35f7da2138650834c97e612a3194"
server_key_version: "encryptionkey@0"
I20260619 18:42:05.132336 22665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:05.155287 22665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:05.158283 22665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:05.158494 22665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:05.165268 22665 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46421
I20260619 18:42:05.165510 22732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46421 every 8 connection(s)
I20260619 18:42:05.166249 22665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:05.168962 22733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:05.172640 22733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:42:05.174217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 22665
I20260619 18:42:05.174326 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:42:05.174594 392 external_mini_cluster.cc:1428] Setting key 376531dd6d1bc0f545b69d99e0211595
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 18:42:05 dist-test-slave-rxv6 krb5kdc[22652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:42:05.189266 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.174855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46421 (local address 127.0.0.1:40648)
0619 18:42:05.175367 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.175420 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:05.176393 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:05.177025 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:05.177030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:05.177450 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:05.177687 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.177700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.178843 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.178846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.179598 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.179607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.180190 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.180194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.180286 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.180861 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:05.180876 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:05.184638 (+ 3762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:05.186482 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.186491 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.186501 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.186788 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.187101 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.187104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.187105 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.187248 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.187925 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:05.187931 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:05.188057 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 18:42:05.188215 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 18:42:05.188472 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":119,"threads_started":1}
I20260619 18:42:05.189270 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.175718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.0.98.62:46421)
0619 18:42:05.176051 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.176062 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.176102 (+ 40us) server_negotiation.cc:407] Connection header received
0619 18:42:05.176677 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.176681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.176852 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.177012 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.177822 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.178736 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.179704 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.180061 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.180206 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.184792 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:05.184798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:05.184812 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:05.184837 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:05.186375 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.186906 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.186910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.186911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.187020 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.187330 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.187333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.187334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.187751 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:05.187938 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:05.188317 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 18:42:05.188588 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":251,"thread_start_us":83,"threads_started":1}
I20260619 18:42:05.191754 22733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:42:05.192392 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.192175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35777 (local address 127.0.0.1:50358)
0619 18:42:05.192278 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.192341 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 18:42:05.192559 22682 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35777: connect: Connection refused (error 111)
I20260619 18:42:05.193275 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.193064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35777 (local address 127.0.0.1:50370)
0619 18:42:05.193167 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.193217 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260619 18:42:05.193418 22733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35777: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35777: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 18:42:05.200073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35777
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46421,127.0.98.61:35777,127.0.98.60:37671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:42:05.242638 22733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } attempt: 1
I20260619 18:42:05.243485 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.243209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35777 (local address 127.0.0.1:50382)
0619 18:42:05.243350 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.243408 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 18:42:05.244182 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.243962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35777 (local address 127.0.0.1:50388)
0619 18:42:05.244078 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.244121 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 18:42:05.244318 22733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35777: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35777: connect: Connection refused (error 111)
W20260619 18:42:05.307108 22740 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:42:05.307391 22740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:05.307426 22740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:05.307489 22740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:05.311443 22740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:05.311496 22740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:05.311522 22740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:05.311542 22740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:05.311568 22740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:05.312492 22733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } attempt: 2
I20260619 18:42:05.313277 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.312998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35777 (local address 127.0.0.1:50390)
0619 18:42:05.313141 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.313204 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 18:42:05.314007 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.313745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35777 (local address 127.0.0.1:50396)
0619 18:42:05.313855 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.313901 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 18:42:05.314145 22733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35777: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35777: connect: Connection refused (error 111)
I20260619 18:42:05.317131 22740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46421,127.0.98.61:35777,127.0.98.60:37671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:05.318216 22740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:05.319191 22740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:05.327601 22745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:05.327663 22740 server_base.cc:1034] running on GCE node
W20260619 18:42:05.327625 22746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:05.327910 22748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:05.328490 22740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:05.329638 22740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:05.330826 22740 hybrid_clock.cc:648] HybridClock initialized: now 1781894525330803 us; error 44 us; skew 500 ppm
Jun 19 18:42:05 dist-test-slave-rxv6 krb5kdc[22652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:05.333524 22740 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 18:42:05.334419 22740 webserver.cc:460] Webserver started at http://127.0.98.61:44445/ using document root <none> and password file <none>
I20260619 18:42:05.334925 22740 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:05.335006 22740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:05.335220 22740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:05.337260 22740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-1/data/instance:
uuid: "d227187cd76046feb46251ae4cbd3916"
format_stamp: "Formatted at 2026-06-19 18:42:05 on dist-test-slave-rxv6"
server_key: "7305de6d270258a5c50eff7c5794a7ee"
server_key_iv: "1953fd93ee99f04037c741d64559e21f"
server_key_version: "encryptionkey@0"
I20260619 18:42:05.337775 22740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-1/wal/instance:
uuid: "d227187cd76046feb46251ae4cbd3916"
format_stamp: "Formatted at 2026-06-19 18:42:05 on dist-test-slave-rxv6"
server_key: "7305de6d270258a5c50eff7c5794a7ee"
server_key_iv: "1953fd93ee99f04037c741d64559e21f"
server_key_version: "encryptionkey@0"
I20260619 18:42:05.341393 22740 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 18:42:05.343922 22755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:05.345129 22740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:05.345252 22740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-1/wal
uuid: "d227187cd76046feb46251ae4cbd3916"
format_stamp: "Formatted at 2026-06-19 18:42:05 on dist-test-slave-rxv6"
server_key: "7305de6d270258a5c50eff7c5794a7ee"
server_key_iv: "1953fd93ee99f04037c741d64559e21f"
server_key_version: "encryptionkey@0"
I20260619 18:42:05.345343 22740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:05.368604 22740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:05.371799 22740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:05.371963 22740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:05.378665 22740 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35777
I20260619 18:42:05.378923 22807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35777 every 8 connection(s)
I20260619 18:42:05.379684 22740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-1/data/info.pb
I20260619 18:42:05.382359 22808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:05.385967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 22740
I20260619 18:42:05.386066 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-1/wal/instance
I20260619 18:42:05.386299 392 external_mini_cluster.cc:1428] Setting key 592ff4470d28728fef24d5567dbe8dc4
I20260619 18:42:05.386165 22808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 18:42:05 dist-test-slave-rxv6 krb5kdc[22652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:42:05.400424 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.388148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40650 (local address 127.0.98.62:46421)
0619 18:42:05.388302 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.388307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.389178 (+ 871us) server_negotiation.cc:407] Connection header received
0619 18:42:05.390225 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.390230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.390295 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.390399 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.391535 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.392342 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.393492 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.393743 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.393827 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.396777 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:05.396780 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:05.396782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:05.396807 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:05.398284 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.398797 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.398800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.398802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.398879 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.399220 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.399224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.399226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.399465 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:05.399580 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:05.400092 (+ 512us) server_negotiation.cc:299] Negotiation successful
0619 18:42:05.400227 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 18:42:05.401129 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.388521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46421 (local address 127.0.0.1:40650)
0619 18:42:05.389007 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.389059 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:05.390022 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:05.390556 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:05.390563 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:05.390897 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:05.391383 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.391397 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.392470 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.392473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.393380 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.393390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.393886 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.393889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.394081 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.394630 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:05.394647 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:05.396609 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:05.398399 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.398405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.398421 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.398702 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.398959 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.398962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.398964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.399107 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.399588 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:05.399594 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:05.399847 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 18:42:05.400004 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 18:42:05.400269 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":123,"threads_started":1}
I20260619 18:42:05.402575 22808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:42:05.405289 22733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } attempt: 3
Jun 19 18:42:05 dist-test-slave-rxv6 krb5kdc[22652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Jun 19 18:42:05 dist-test-slave-rxv6 krb5kdc[22652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 18:42:05.411077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37671
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46421,127.0.98.61:35777,127.0.98.60:37671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:42:05.412393 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.403060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35777 (local address 127.0.0.1:50404)
0619 18:42:05.403172 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.403186 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:05.403258 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:05.403862 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:05.403864 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:05.403993 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:05.404091 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.404097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.405045 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.405047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.405718 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.405728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.406474 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.406478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.406616 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.407142 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:05.407154 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:05.408500 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:05.410776 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.410779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.410781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.411007 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.411321 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.411324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.411326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.411411 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.411915 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:05.411917 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:05.411990 (+ 73us) client_negotiation.cc:769] Sending connection context
0619 18:42:05.412074 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 18:42:05.412204 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":39}
I20260619 18:42:05.412979 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.403196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50404 (local address 127.0.98.61:35777)
0619 18:42:05.403506 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.403514 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.403532 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:42:05.403568 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.403572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.403749 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.403861 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.404190 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.404932 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.405828 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.406378 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.406469 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.408663 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:05.408672 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:05.408683 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:05.408703 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:05.410650 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.411124 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.411131 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.411133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.411224 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.411489 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.411492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.411494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.411769 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:05.411885 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:05.412639 (+ 754us) server_negotiation.cc:299] Negotiation successful
0619 18:42:05.412814 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":226,"thread_start_us":140,"threads_started":1}
I20260619 18:42:05.415010 22808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:42:05.415653 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.405712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35777 (local address 127.0.0.1:50410)
0619 18:42:05.405837 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.405852 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:05.405954 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:05.406270 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:05.406273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:05.406416 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:05.406527 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.406532 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.407414 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.407417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.408670 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.408679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.409227 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.409232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.409345 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.409842 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:05.409853 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:05.411150 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:05.414171 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.414174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.414177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.414403 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.414687 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.414690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.414692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.414774 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.415152 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:05.415155 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:05.415248 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 18:42:05.415336 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 18:42:05.415498 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 18:42:05.415745 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.415453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37671 (local address 127.0.0.1:57162)
0619 18:42:05.415589 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.415669 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 18:42:05.416131 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.405784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50410 (local address 127.0.98.61:35777)
0619 18:42:05.406075 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.406083 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.406095 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:42:05.406124 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.406127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.406165 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.406240 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.406647 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.407302 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.408760 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.409047 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.409154 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.411259 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:05.411262 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:05.411265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:05.411286 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:05.414074 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.414499 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.414503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.414504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.414601 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.414871 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.414874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.414876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.415055 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:05.415142 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:05.415854 (+ 712us) server_negotiation.cc:299] Negotiation successful
0619 18:42:05.415966 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":113,"threads_started":1}
W20260619 18:42:05.416177 22757 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
I20260619 18:42:05.417024 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.416810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37671 (local address 127.0.0.1:57176)
0619 18:42:05.416913 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.416963 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 18:42:05.417064 22733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } has no permanent_uuid. Determining permanent_uuid...
W20260619 18:42:05.417392 22808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37671: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
I20260619 18:42:05.417681 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.417470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37671 (local address 127.0.0.1:57190)
0619 18:42:05.417573 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.417621 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 18:42:05.418309 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.418089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37671 (local address 127.0.0.1:57198)
0619 18:42:05.418216 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.418260 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 18:42:05.418416 22733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37671: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
I20260619 18:42:05.449559 22733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } attempt: 1
I20260619 18:42:05.450382 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.450107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37671 (local address 127.0.0.1:57202)
0619 18:42:05.450236 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.450299 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 18:42:05.451062 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.450853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37671 (local address 127.0.0.1:57204)
0619 18:42:05.450961 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.451006 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 18:42:05.451162 22733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37671: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
I20260619 18:42:05.466648 22808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } attempt: 1
I20260619 18:42:05.467442 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.467150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37671 (local address 127.0.0.1:57206)
0619 18:42:05.467295 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.467359 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 18:42:05.468108 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.467909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37671 (local address 127.0.0.1:57222)
0619 18:42:05.467997 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.468046 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 18:42:05.468225 22808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37671: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
W20260619 18:42:05.519941 22815 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:42:05.520196 22815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:05.520217 22815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:05.520252 22815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:05.524099 22815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:05.524151 22815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:05.524168 22815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:05.524183 22815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:05.524196 22815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:05.526330 22733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } attempt: 2
I20260619 18:42:05.527100 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.526794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37671 (local address 127.0.0.1:57230)
0619 18:42:05.526944 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.527004 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 18:42:05.527818 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.527611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37671 (local address 127.0.0.1:57236)
0619 18:42:05.527712 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.527762 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 18:42:05.527944 22733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37671: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
I20260619 18:42:05.529536 22815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46421,127.0.98.61:35777,127.0.98.60:37671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:05.530534 22815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:05.531452 22815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:42:05.536355 22808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } attempt: 2
I20260619 18:42:05.537053 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.536808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37671 (local address 127.0.0.1:57242)
0619 18:42:05.536931 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.536983 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 18:42:05.537698 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.537466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37671 (local address 127.0.0.1:57244)
0619 18:42:05.537594 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.537642 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 18:42:05.537846 22808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37671: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37671: connect: Connection refused (error 111)
W20260619 18:42:05.539790 22824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:05.539803 22821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:05.539819 22822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:05.539958 22815 server_base.cc:1034] running on GCE node
I20260619 18:42:05.540539 22815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:05.541594 22815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:05.542742 22815 hybrid_clock.cc:648] HybridClock initialized: now 1781894525542735 us; error 68 us; skew 500 ppm
Jun 19 18:42:05 dist-test-slave-rxv6 krb5kdc[22652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:05.545307 22815 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 18:42:05.546185 22815 webserver.cc:460] Webserver started at http://127.0.98.60:46739/ using document root <none> and password file <none>
I20260619 18:42:05.546680 22815 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:05.546734 22815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:05.546916 22815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:05.548985 22815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-2/data/instance:
uuid: "802df88b22024f9bb27216047dccf13b"
format_stamp: "Formatted at 2026-06-19 18:42:05 on dist-test-slave-rxv6"
server_key: "1536ccec8cea9906c474761dc9b89c32"
server_key_iv: "addd5abe79a1075ee58f65615fad2211"
server_key_version: "encryptionkey@0"
I20260619 18:42:05.549482 22815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-2/wal/instance:
uuid: "802df88b22024f9bb27216047dccf13b"
format_stamp: "Formatted at 2026-06-19 18:42:05 on dist-test-slave-rxv6"
server_key: "1536ccec8cea9906c474761dc9b89c32"
server_key_iv: "addd5abe79a1075ee58f65615fad2211"
server_key_version: "encryptionkey@0"
I20260619 18:42:05.552971 22815 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:42:05.555270 22831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:05.556411 22815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:42:05.556507 22815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-2/wal
uuid: "802df88b22024f9bb27216047dccf13b"
format_stamp: "Formatted at 2026-06-19 18:42:05 on dist-test-slave-rxv6"
server_key: "1536ccec8cea9906c474761dc9b89c32"
server_key_iv: "addd5abe79a1075ee58f65615fad2211"
server_key_version: "encryptionkey@0"
I20260619 18:42:05.556588 22815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:05.580652 22815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:05.583833 22815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:05.584002 22815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:05.590662 22815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37671
I20260619 18:42:05.590909 22883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37671 every 8 connection(s)
I20260619 18:42:05.591591 22815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-2/data/info.pb
I20260619 18:42:05.594264 22884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:05.597822 22884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:42:05.598276 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 22815
I20260619 18:42:05.598395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781894249862417-392-0/minicluster-data/master-2/wal/instance
I20260619 18:42:05.598682 392 external_mini_cluster.cc:1428] Setting key 3f1ce6c6a6c0b32cee5e5c37e392b618
Jun 19 18:42:05 dist-test-slave-rxv6 krb5kdc[22652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 18:42:05 dist-test-slave-rxv6 krb5kdc[22652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 18:42:05.610721 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.599807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40652 (local address 127.0.98.62:46421)
0619 18:42:05.599930 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.599934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.600536 (+ 602us) server_negotiation.cc:407] Connection header received
0619 18:42:05.601385 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.601387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.601437 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.601509 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.602735 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.603263 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.604214 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.604431 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.604516 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.606886 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:05.606889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:05.606892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:05.606922 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:05.608470 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.608960 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.608964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.608966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.609057 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.609382 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.609385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.609387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.609664 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:05.609746 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:05.610266 (+ 520us) server_negotiation.cc:299] Negotiation successful
0619 18:42:05.610389 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:42:05.611342 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.599902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46421 (local address 127.0.0.1:40652)
0619 18:42:05.600378 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.600425 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:05.601168 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:05.602030 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:05.602034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:05.602339 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:05.602600 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.602611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.603375 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.603378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.604100 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.604109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.604560 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.604563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.604687 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.605260 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:05.605280 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:05.606728 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:05.608576 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.608583 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.608591 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.608864 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.609138 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.609141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.609143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.609301 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.609758 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:05.609763 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:05.610035 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 18:42:05.610192 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 18:42:05.610509 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":51,"spinlock_wait_cycles":7424,"thread_start_us":125,"threads_started":1}
I20260619 18:42:05.611346 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.599910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57248 (local address 127.0.98.60:37671)
0619 18:42:05.600370 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.600387 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.600423 (+ 36us) server_negotiation.cc:407] Connection header received
0619 18:42:05.601154 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.601178 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.601409 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.601567 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.602388 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.603333 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.604149 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.604493 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.604661 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.607278 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:05.607283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:05.607292 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:05.607312 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:05.608979 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.609430 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.609436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.609437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.609510 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.609737 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.609740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.609742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.610037 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:05.610189 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:05.610404 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 18:42:05.610557 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":361,"spinlock_wait_cycles":75008,"thread_start_us":141,"threads_started":1}
I20260619 18:42:05.612562 22884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:42:05 dist-test-slave-rxv6 krb5kdc[22652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 18:42:05 dist-test-slave-rxv6 krb5kdc[22652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:42:05.620155 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.612982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35777 (local address 127.0.0.1:50412)
0619 18:42:05.613084 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.613097 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:05.613283 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:05.613550 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:05.613553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:05.613706 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:05.613805 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.613812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.614611 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.614613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.615115 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.615123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.615672 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.615674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.615739 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.616121 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:05.616131 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:05.617274 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:05.618865 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.618868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.618870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.619069 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.619313 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.619316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.619318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.619402 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.619778 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:05.619780 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:05.619852 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 18:42:05.619929 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 18:42:05.620045 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 18:42:05.620263 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.613095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50412 (local address 127.0.98.61:35777)
0619 18:42:05.613206 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.613211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.613224 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:42:05.613391 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.613394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.613448 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.613523 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.613981 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.614521 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.615199 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.615581 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.615679 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.617422 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:05.617425 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:05.617427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:05.617451 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:05.618770 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.619155 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.619159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.619161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.619229 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.619481 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.619484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.619486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.619688 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:05.619782 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:05.619997 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 18:42:05.620108 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 18:42:05.620826 22884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:42:05.622100 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.613618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.0.98.62:46421)
0619 18:42:05.613719 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.613723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.613757 (+ 34us) server_negotiation.cc:407] Connection header received
0619 18:42:05.613855 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.613857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.613893 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.613983 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.615140 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.615808 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.616498 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.616719 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.616798 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.618948 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:05.618951 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:05.618953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:05.618979 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:05.620391 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.620766 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.620769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.620770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.620849 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.621106 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.621109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.621110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.621286 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:05.621520 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:05.621808 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 18:42:05.621987 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Jun 19 18:42:05 dist-test-slave-rxv6 krb5kdc[22652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 18:42:05.627091 22733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } attempt: 3
Jun 19 18:42:05 dist-test-slave-rxv6 krb5kdc[22652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 18:42:05.628547 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.621415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57252 (local address 127.0.98.60:37671)
0619 18:42:05.621503 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.621507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.621519 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:42:05.621633 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.621635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.621670 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.621725 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.622124 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.622755 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.623551 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.623909 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.624008 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.625744 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:05.625747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:05.625750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:05.625772 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:05.627008 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.627420 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.627423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.627425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.627502 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.627779 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.627782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.627784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.627972 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:05.628057 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:05.628290 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 18:42:05.628384 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":43}
I20260619 18:42:05.628716 22887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.621313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37671 (local address 127.0.0.1:57252)
0619 18:42:05.621469 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.621482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:05.621548 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:05.621757 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:05.621760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:05.621932 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:05.622024 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.622030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.622849 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.622852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.623467 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.623476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.624017 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.624020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.624091 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.624462 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:05.624473 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:05.625633 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:05.627115 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.627118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.627120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.627329 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.627594 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.627596 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.627598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.627685 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.628087 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:05.628089 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:05.628164 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 18:42:05.628254 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 18:42:05.628595 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":3}
I20260619 18:42:05.629009 22808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } attempt: 3
I20260619 18:42:05.631305 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.623122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.0.98.61:35777)
0619 18:42:05.623215 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.623219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.623230 (+ 11us) server_negotiation.cc:407] Connection header received
0619 18:42:05.623335 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.623338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.623375 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.623435 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.624093 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.624565 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.625491 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.625715 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.625796 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.628081 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:05.628085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:05.628087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:05.628109 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:05.629704 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.630183 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.630186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.630188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.630271 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.630531 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.630534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.630535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.630722 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:05.630922 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:05.631071 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 18:42:05.631174 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 18:42:05.632117 22884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b: Bootstrap starting.
Jun 19 18:42:05 dist-test-slave-rxv6 krb5kdc[22652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 18:42:05.634606 22884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:05.635673 22884 log.cc:824] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:05.636880 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.627446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37671 (local address 127.0.0.1:57268)
0619 18:42:05.627699 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.627712 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:05.627779 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:05.628183 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:05.628186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:05.628339 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:05.628432 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.628439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.630846 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.630849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.631576 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.631586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.631996 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.631999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.632077 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.632462 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:05.632475 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:05.633476 (+ 1001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:05.635326 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.635329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.635331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.635566 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.635874 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.635878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.635880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.635980 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.636422 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:05.636426 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:05.636521 (+ 95us) client_negotiation.cc:769] Sending connection context
0619 18:42:05.636603 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 18:42:05.636733 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260619 18:42:05.637367 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.627649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57268 (local address 127.0.98.60:37671)
0619 18:42:05.627990 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.627999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.628009 (+ 10us) server_negotiation.cc:407] Connection header received
0619 18:42:05.628037 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.628039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.628076 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.628151 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.628841 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.630727 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.631691 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.631902 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.631978 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.633592 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:05.633596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:05.633598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:05.633620 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:05.635209 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.635683 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.635686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.635688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.635771 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.636092 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.636094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.636096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.636316 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:05.637042 (+ 726us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:05.637115 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 18:42:05.637218 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":118,"threads_started":1}
I20260619 18:42:05.638226 22884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b: No bootstrap required, opened a new log
Jun 19 18:42:05 dist-test-slave-rxv6 krb5kdc[22652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 18:42:05.641566 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:42:05.643226 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.629363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37671 (local address 127.0.0.1:57284)
0619 18:42:05.629478 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:05.629492 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:05.629560 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:05.634419 (+ 4859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:05.634422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:05.634588 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:05.634701 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.634708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.635625 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.635629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.636355 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:05.636365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.637667 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.637671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:05.637755 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.638169 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:05.638210 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:05.639613 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:05.641559 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.641562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.641564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.641801 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.642163 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:05.642167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:05.642169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:05.642255 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:05.642724 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:05.642727 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:05.642845 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 18:42:05.642925 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 18:42:05.643063 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 18:42:05.644441 22884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } }
I20260619 18:42:05.643518 22707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b8b1cca3ad0948429fdc02a995a40433"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40656
I20260619 18:42:05.644906 22884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 802df88b22024f9bb27216047dccf13b, State: Initialized, Role: FOLLOWER
I20260619 18:42:05.645255 22884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } }
I20260619 18:42:05.645583 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.629580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57284 (local address 127.0.98.60:37671)
0619 18:42:05.629664 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.629668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.629678 (+ 10us) server_negotiation.cc:407] Connection header received
0619 18:42:05.629705 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.629707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.634298 (+ 4591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.634800 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.634893 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.635508 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.636488 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.637548 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.637656 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.639750 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:05.639754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:05.639756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:05.639778 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:05.641440 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.641968 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.641970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.641972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.642055 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.642363 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.642366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.642367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.642596 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:05.645190 (+ 2594us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:05.645320 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 18:42:05.645447 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 18:42:05.646281 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } } }
I20260619 18:42:05.646414 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:05.646873 22884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:05.648357 22733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433: Bootstrap starting.
I20260619 18:42:05.649138 22808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916: Bootstrap starting.
I20260619 18:42:05.651729 22808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916: Neither blocks nor log segments found. Creating new log.
W20260619 18:42:05.653136 22902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:42:05.653203 22902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:42:05.654467 22733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:05.654659 22808 log.cc:824] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:05.656037 22733 log.cc:824] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:05.653287 22815 master_runner.cc:186] Error getting master registration for 127.0.98.62:46421: OK, instance_id { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" instance_seqno: 1781894525163368 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:42:05.656636 22815 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:42:05.658597 22733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433: No bootstrap required, opened a new log
I20260619 18:42:05.659001 22808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916: No bootstrap required, opened a new log
I20260619 18:42:05.661623 22733 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } }
I20260619 18:42:05.661830 22733 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8b1cca3ad0948429fdc02a995a40433, State: Initialized, Role: FOLLOWER
I20260619 18:42:05.662165 22733 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } }
I20260619 18:42:05.662629 22906 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } } }
I20260619 18:42:05.662842 22906 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:05.663071 22808 raft_consensus.cc:348] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } }
I20260619 18:42:05.663275 22808 raft_consensus.cc:729] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d227187cd76046feb46251ae4cbd3916, State: Initialized, Role: FOLLOWER
I20260619 18:42:05.663444 22733 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:05.663518 22707 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 18:42:05.663609 22707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:05.663568 22808 consensus_queue.cc:260] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } }
I20260619 18:42:05.664781 22808 sys_catalog.cc:564] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:05.664846 22707 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } }
I20260619 18:42:05.665751 22707 leader_election.cc:288] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [CANDIDATE]: Term 1 election: Requested vote from peers d227187cd76046feb46251ae4cbd3916 (127.0.98.61:35777), 802df88b22024f9bb27216047dccf13b (127.0.98.60:37671)
I20260619 18:42:05.667342 22858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8b1cca3ad0948429fdc02a995a40433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "802df88b22024f9bb27216047dccf13b"
I20260619 18:42:05.667584 22858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:05.667945 22782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b8b1cca3ad0948429fdc02a995a40433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d227187cd76046feb46251ae4cbd3916"
I20260619 18:42:05.668160 22782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:05.668840 22858 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8b1cca3ad0948429fdc02a995a40433 in term 1.
I20260619 18:42:05.669237 22682 leader_election.cc:302] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 802df88b22024f9bb27216047dccf13b, b8b1cca3ad0948429fdc02a995a40433; no voters:
I20260619 18:42:05.669384 22782 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8b1cca3ad0948429fdc02a995a40433 in term 1.
I20260619 18:42:05.669973 22906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:05.670161 22906 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [term 1 LEADER]: Becoming Leader. State: Replica: b8b1cca3ad0948429fdc02a995a40433, State: Running, Role: LEADER
I20260619 18:42:05.670454 22906 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } }
I20260619 18:42:05.672237 22906 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b1cca3ad0948429fdc02a995a40433. Latest consensus state: current_term: 1 leader_uuid: "b8b1cca3ad0948429fdc02a995a40433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } } }
I20260619 18:42:05.672330 22906 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:05.670894 22907 sys_catalog.cc:455] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } } }
I20260619 18:42:05.673163 22907 sys_catalog.cc:458] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:05.678473 22665 master_runner.cc:263] No leader master found from master b8b1cca3ad0948429fdc02a995a40433
I20260619 18:42:05.678545 22665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 18:42:05.682714 22933 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:42:05.682864 22933 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:42:05.685863 22814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.675509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.0.98.61:35777)
0619 18:42:05.675611 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.675615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.677133 (+ 1518us) server_negotiation.cc:407] Connection header received
0619 18:42:05.677304 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.677307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.677352 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.677397 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.678362 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.679110 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.680243 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.680548 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.680631 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.682488 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:05.682492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:05.682494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:05.682519 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:05.684115 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.684558 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.684562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.684564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.684644 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.684931 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.684935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.684937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.685150 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:05.685261 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:05.685585 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 18:42:05.685693 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 18:42:05.694409 22918 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:42:05.694584 22918 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:42:05.694653 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.676130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57288 (local address 127.0.98.60:37671)
0619 18:42:05.676234 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.676238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.686421 (+ 10183us) server_negotiation.cc:407] Connection header received
0619 18:42:05.686574 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.686577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.686627 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.686673 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.687492 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.688156 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.689168 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.689483 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.689554 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.691188 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:05.691193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:05.691196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:05.691224 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:05.692835 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.693307 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.693310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.693312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.693399 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.693686 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.693690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.693692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.693949 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:05.694048 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:05.694386 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 18:42:05.694503 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:42:05.699864 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:05.676380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40660 (local address 127.0.98.62:46421)
0619 18:42:05.676483 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.676487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.683278 (+ 6791us) server_negotiation.cc:407] Connection header received
0619 18:42:05.683472 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.683475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.683527 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.683579 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.684360 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.685053 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.686312 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:05.686612 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:05.686681 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:05.696348 (+ 9667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:05.696353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:05.696356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:05.696384 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:05.698022 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.698500 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.698504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.698505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.698591 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:05.698893 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:05.698897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:05.698899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:05.699110 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:05.699216 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:05.699570 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 18:42:05.699690 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:42:05.701126 22935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:05.703303 22935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:05.717626 22858 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b [term 1 FOLLOWER]: Refusing update from remote peer b8b1cca3ad0948429fdc02a995a40433: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:05.717628 22782 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916 [term 1 FOLLOWER]: Refusing update from remote peer b8b1cca3ad0948429fdc02a995a40433: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:05.718341 22929 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:05.718542 22906 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:05.725358 22907 sys_catalog.cc:455] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b1cca3ad0948429fdc02a995a40433. Latest consensus state: current_term: 1 leader_uuid: "b8b1cca3ad0948429fdc02a995a40433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } } }
I20260619 18:42:05.725543 22907 sys_catalog.cc:458] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:05.726496 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b1cca3ad0948429fdc02a995a40433. Latest consensus state: current_term: 1 leader_uuid: "b8b1cca3ad0948429fdc02a995a40433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } } }
I20260619 18:42:05.726684 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:05.738338 22906 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8b1cca3ad0948429fdc02a995a40433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } } }
I20260619 18:42:05.738481 22906 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:05.739022 22929 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b8b1cca3ad0948429fdc02a995a40433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } } }
I20260619 18:42:05.739130 22929 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:05.740159 22891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8b1cca3ad0948429fdc02a995a40433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } } }
I20260619 18:42:05.740144 22907 sys_catalog.cc:455] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b8b1cca3ad0948429fdc02a995a40433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8b1cca3ad0948429fdc02a995a40433" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46421 } } peers { permanent_uuid: "d227187cd76046feb46251ae4cbd3916" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35777 } } peers { permanent_uuid: "802df88b22024f9bb27216047dccf13b" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37671 } } }
I20260619 18:42:05.740265 22907 sys_catalog.cc:458] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:05.740265 22891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:05.740650 22944 mvcc.cc:205] Tried to move back new op lower bound from 7298639977331654656 to 7298639977149915136. Current Snapshot: MvccSnapshot[applied={T|T < 7298639977331654656}]
I20260619 18:42:05.746984 22935 catalog_manager.cc:1347] Generated new cluster ID: c87c49eaaaf24168902f993af0a30dd4
I20260619 18:42:05.747061 22935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:05.770921 22935 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:05.772210 22935 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:05.786346 22935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8b1cca3ad0948429fdc02a995a40433: Generated new TSK 0
I20260619 18:42:05.787254 22935 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:42:05.817140 22814 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:05.814640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.0.98.61:35777)
0619 18:42:05.814772 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.814777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.816067 (+ 1290us) server_negotiation.cc:407] Connection header received
0619 18:42:05.816264 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.816268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.816321 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.816373 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.816999 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50434: BlockingRecv error: recv got EOF from 127.0.0.1:50434 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 18:42:05.818553 22888 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:05.815090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57302 (local address 127.0.98.60:37671)
0619 18:42:05.815207 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.815212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.817620 (+ 2408us) server_negotiation.cc:407] Connection header received
0619 18:42:05.817797 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.817801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.817847 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.817896 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.818438 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57302: BlockingRecv error: recv got EOF from 127.0.0.1:57302 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 18:42:05.819825 22737 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:05.815458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40672 (local address 127.0.98.62:46421)
0619 18:42:05.815567 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.815571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.818978 (+ 3407us) server_negotiation.cc:407] Connection header received
0619 18:42:05.819134 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.819137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.819187 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.819234 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.819692 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40672: BlockingRecv error: recv got EOF from 127.0.0.1:40672 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 18:42:05.824218 22814 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:05.822080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.0.98.61:35777)
0619 18:42:05.822185 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.822189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.823279 (+ 1090us) server_negotiation.cc:407] Connection header received
0619 18:42:05.823492 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.823495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.823568 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.823615 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.824105 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50448: BlockingRecv error: recv got EOF from 127.0.0.1:50448 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 18:42:05.829272 22814 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:05.827766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50452 (local address 127.0.98.61:35777)
0619 18:42:05.827872 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.827876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.828397 (+ 521us) server_negotiation.cc:407] Connection header received
0619 18:42:05.828574 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.828577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.828629 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.828677 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.829168 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50452: BlockingRecv error: recv got EOF from 127.0.0.1:50452 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 18:42:05.834880 22814 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:05.833430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.0.98.61:35777)
0619 18:42:05.833525 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.833529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.834033 (+ 504us) server_negotiation.cc:407] Connection header received
0619 18:42:05.834204 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.834207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.834256 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.834304 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.834776 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50458: BlockingRecv error: recv got EOF from 127.0.0.1:50458 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 18:42:05.843600 22814 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:05.841893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50470 (local address 127.0.98.61:35777)
0619 18:42:05.842101 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.842105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.842642 (+ 537us) server_negotiation.cc:407] Connection header received
0619 18:42:05.842882 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.842886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.842940 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.842991 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.843490 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50470: BlockingRecv error: recv got EOF from 127.0.0.1:50470 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
W20260619 18:42:05.855919 22814 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:05.854173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.0.98.61:35777)
0619 18:42:05.854294 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.854299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.854917 (+ 618us) server_negotiation.cc:407] Connection header received
0619 18:42:05.855116 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.855120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.855172 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.855222 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.855799 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50484: BlockingRecv error: recv got EOF from 127.0.0.1:50484 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260619 18:42:05.876034 22814 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:05.874620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50500 (local address 127.0.98.61:35777)
0619 18:42:05.874759 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.874764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.875136 (+ 372us) server_negotiation.cc:407] Connection header received
0619 18:42:05.875276 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.875280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.875342 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.875438 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.875911 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50500: BlockingRecv error: recv got EOF from 127.0.0.1:50500 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 18:42:05.911675 22814 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:05.910423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.0.98.61:35777)
0619 18:42:05.910881 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.910887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.910901 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:42:05.910955 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.910957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.911000 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.911073 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.911569 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50514: BlockingRecv error: recv got EOF from 127.0.0.1:50514 (error 108)
Metrics: {"server-negotiator.queue_time_us":361}
W20260619 18:42:05.978863 22814 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:05.977765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50516 (local address 127.0.98.61:35777)
0619 18:42:05.977947 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:05.977952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:05.978036 (+ 84us) server_negotiation.cc:407] Connection header received
0619 18:42:05.978189 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:05.978191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:05.978239 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:05.978323 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:05.978758 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50516: BlockingRecv error: recv got EOF from 127.0.0.1:50516 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 18:42:06.110078 22814 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:06.108949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50518 (local address 127.0.98.61:35777)
0619 18:42:06.109098 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:06.109103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:06.109197 (+ 94us) server_negotiation.cc:407] Connection header received
0619 18:42:06.109353 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:06.109355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:06.109403 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:06.109474 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:06.109980 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50518: BlockingRecv error: recv got EOF from 127.0.0.1:50518 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 18:42:06.369299 22814 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:06.368243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50532 (local address 127.0.98.61:35777)
0619 18:42:06.368390 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:06.368394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:06.368478 (+ 84us) server_negotiation.cc:407] Connection header received
0619 18:42:06.368606 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:06.368610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:06.368657 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:06.368732 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:06.369197 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50532: BlockingRecv error: recv got EOF from 127.0.0.1:50532 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 18:42:06.654603 22902 catalog_manager.cc:1259] Loaded cluster ID: c87c49eaaaf24168902f993af0a30dd4
I20260619 18:42:06.654686 22902 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b: loading cluster ID for follower catalog manager: success
I20260619 18:42:06.656344 22902 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b: acquiring CA information for follower catalog manager: success
I20260619 18:42:06.657475 22902 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 802df88b22024f9bb27216047dccf13b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:42:06.684381 22933 catalog_manager.cc:1259] Loaded cluster ID: c87c49eaaaf24168902f993af0a30dd4
I20260619 18:42:06.684479 22933 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916: loading cluster ID for follower catalog manager: success
I20260619 18:42:06.686197 22933 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916: acquiring CA information for follower catalog manager: success
I20260619 18:42:06.687392 22933 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d227187cd76046feb46251ae4cbd3916: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:42:06.887104 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:06.883369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.0.98.61:35777)
0619 18:42:06.883581 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:06.883584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:06.883632 (+ 48us) server_negotiation.cc:407] Connection header received
0619 18:42:06.883741 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:06.883744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:06.883881 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:06.883984 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:06.884179 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:06.884810 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:06.885638 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:06.885868 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:06.885995 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:06.886246 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:06.886692 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:06.886791 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:06.886894 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 18:42:06.886946 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":73,"threads_started":1}
I20260619 18:42:06.893661 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:06.890173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57304 (local address 127.0.98.60:37671)
0619 18:42:06.890499 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:06.890502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:06.890515 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:42:06.890592 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:06.890594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:06.890705 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:06.890798 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:06.890969 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:06.891491 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:06.892294 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:06.892544 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:06.892624 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:06.892880 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:06.893287 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:06.893401 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:06.893456 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 18:42:06.893506 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":137,"threads_started":1}
I20260619 18:42:06.900530 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:06.896764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57312 (local address 127.0.98.60:37671)
0619 18:42:06.896880 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:06.896884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:06.897151 (+ 267us) server_negotiation.cc:407] Connection header received
0619 18:42:06.897275 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:06.897279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:06.897334 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:06.897392 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:06.897692 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:06.898432 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:06.899560 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:06.899768 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:06.899875 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:06.900098 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:06.900170 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:06.900223 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:06.900355 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 18:42:06.900407 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:42:06.901118 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:06.896361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.0.98.61:35777)
0619 18:42:06.896494 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:06.896499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:06.896907 (+ 408us) server_negotiation.cc:407] Connection header received
0619 18:42:06.897061 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:06.897065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:06.897108 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:06.897168 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:06.897442 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:06.898087 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:06.899984 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:06.900281 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:06.900367 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:06.900688 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:06.900761 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:06.900815 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:06.900965 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 18:42:06.901023 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:42:06.901202 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:06.896390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40674 (local address 127.0.98.62:46421)
0619 18:42:06.896630 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:06.896634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:06.896862 (+ 228us) server_negotiation.cc:407] Connection header received
0619 18:42:06.897024 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:06.897027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:06.897196 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:06.897304 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:06.897489 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:06.898323 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:06.899603 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:06.899957 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:06.900045 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:06.900310 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:06.900626 (+ 316us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:06.900978 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:06.901011 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 18:42:06.901069 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":82,"threads_started":1}
I20260619 18:42:06.903828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 22665
I20260619 18:42:06.909178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 22740
I20260619 18:42:06.913894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 22815
2026-06-19T18:42:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2990 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 18:42:06 dist-test-slave-rxv6 krb5kdc[23033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 18:42:06 dist-test-slave-rxv6 krb5kdc[23033](info): set up 2 sockets
Jun 19 18:42:06 dist-test-slave-rxv6 krb5kdc[23033](info): commencing operation
krb5kdc: starting...
W20260619 18:42:07.870093 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/test-user.keytab.
Jun 19 18:42:07 dist-test-slave-rxv6 krb5kdc[23033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T18:42:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T18:42:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 18:42:07.980949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35977
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35977,127.0.98.61:44607,127.0.98.60:46393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 18:42:08.088950 23046 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:42:08.089257 23046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:08.089283 23046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:08.089344 23046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:08.093345 23046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:08.093405 23046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:08.093423 23046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:08.093438 23046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:08.093465 23046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:08.098740 23046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35977,127.0.98.61:44607,127.0.98.60:46393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:08.099771 23046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:08.100755 23046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:08.108970 23054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:08.108969 23051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:08.108964 23052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:08.109098 23046 server_base.cc:1034] running on GCE node
I20260619 18:42:08.109795 23046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:08.110890 23046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:08.112051 23046 hybrid_clock.cc:648] HybridClock initialized: now 1781894528112025 us; error 37 us; skew 500 ppm
Jun 19 18:42:08 dist-test-slave-rxv6 krb5kdc[23033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:08.120709 23046 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 18:42:08.121737 23046 webserver.cc:460] Webserver started at http://127.0.98.62:45283/ using document root <none> and password file <none>
I20260619 18:42:08.122314 23046 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:08.122382 23046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:08.122587 23046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:08.124603 23046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-0/data/instance:
uuid: "d9f1582d9d0140339c83d5ecafe1d555"
format_stamp: "Formatted at 2026-06-19 18:42:08 on dist-test-slave-rxv6"
server_key: "fadf88674c08e618d0b3daaf2ace51a7"
server_key_iv: "30292530fadfd5f6bd6600fc11e1bfbf"
server_key_version: "encryptionkey@0"
I20260619 18:42:08.125149 23046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-0/wal/instance:
uuid: "d9f1582d9d0140339c83d5ecafe1d555"
format_stamp: "Formatted at 2026-06-19 18:42:08 on dist-test-slave-rxv6"
server_key: "fadf88674c08e618d0b3daaf2ace51a7"
server_key_iv: "30292530fadfd5f6bd6600fc11e1bfbf"
server_key_version: "encryptionkey@0"
I20260619 18:42:08.128860 23046 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:42:08.131219 23061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:08.132411 23046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:42:08.132524 23046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-0/wal
uuid: "d9f1582d9d0140339c83d5ecafe1d555"
format_stamp: "Formatted at 2026-06-19 18:42:08 on dist-test-slave-rxv6"
server_key: "fadf88674c08e618d0b3daaf2ace51a7"
server_key_iv: "30292530fadfd5f6bd6600fc11e1bfbf"
server_key_version: "encryptionkey@0"
I20260619 18:42:08.132614 23046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:08.142004 23046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:08.145056 23046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:08.145227 23046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:08.152032 23046 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35977
I20260619 18:42:08.152277 23113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35977 every 8 connection(s)
I20260619 18:42:08.152948 23046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-0/data/info.pb
I20260619 18:42:08.155632 23114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:08.156575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 23046
I20260619 18:42:08.156698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-0/wal/instance
I20260619 18:42:08.156975 392 external_mini_cluster.cc:1428] Setting key d0f5a24d6622cc32fa99f08500e47b8d
I20260619 18:42:08.159351 23114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 18:42:08 dist-test-slave-rxv6 krb5kdc[23033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:42:08.175024 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.161594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35977 (local address 127.0.0.1:54886)
0619 18:42:08.162175 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.162211 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:08.163004 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:08.163675 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:08.163681 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:08.164934 (+ 1253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:08.165376 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.165392 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.166581 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.166583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.167349 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.167358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.167998 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.168002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.168157 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.168675 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:08.168691 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:08.170222 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:08.171910 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.171918 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.171928 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.172129 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.172413 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.172416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.172418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.172522 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.173059 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:08.173065 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:08.173194 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 18:42:08.173430 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 18:42:08.173976 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":87,"spinlock_wait_cycles":25984,"thread_start_us":152,"threads_started":1}
I20260619 18:42:08.175024 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.161593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.0.98.62:35977)
0619 18:42:08.162040 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.162058 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.162314 (+ 256us) server_negotiation.cc:407] Connection header received
0619 18:42:08.163298 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.163302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.163492 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.163656 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.165525 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.166468 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.167451 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.167853 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.168010 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.170345 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:08.170351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:08.170360 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:08.170380 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:08.171814 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.172222 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.172227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.172228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.172332 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.172598 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.172601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.172603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.172960 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:08.173074 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:08.173530 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 18:42:08.173861 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":309,"thread_start_us":126,"threads_started":1}
I20260619 18:42:08.177503 23114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:42:08.178608 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.178211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44607 (local address 127.0.0.1:53544)
0619 18:42:08.178395 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.178495 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260619 18:42:08.178818 23063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44607: connect: Connection refused (error 111)
I20260619 18:42:08.179706 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.179490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44607 (local address 127.0.0.1:53554)
0619 18:42:08.179604 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.179651 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 18:42:08.179905 23114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44607: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44607: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 18:42:08.184396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44607
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35977,127.0.98.61:44607,127.0.98.60:46393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:42:08.229159 23114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } attempt: 1
I20260619 18:42:08.230216 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.229820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44607 (local address 127.0.0.1:53566)
0619 18:42:08.230010 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.230113 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 18:42:08.230959 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.230747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44607 (local address 127.0.0.1:53568)
0619 18:42:08.230843 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.230896 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 18:42:08.231076 23114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44607: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44607: connect: Connection refused (error 111)
W20260619 18:42:08.295426 23121 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:42:08.295727 23121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:08.295749 23121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:08.295787 23121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 18:42:08.299243 23114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } attempt: 2
W20260619 18:42:08.299907 23121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:08.299969 23121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:08.299995 23121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:08.300017 23121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:08.300045 23121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:08.300161 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.299802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44607 (local address 127.0.0.1:53576)
0619 18:42:08.299971 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.300073 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 18:42:08.300840 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.300620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44607 (local address 127.0.0.1:53590)
0619 18:42:08.300715 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.300767 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 18:42:08.300966 23114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:44607: Network error: Client connection negotiation failed: client connection to 127.0.98.61:44607: connect: Connection refused (error 111)
I20260619 18:42:08.305576 23121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35977,127.0.98.61:44607,127.0.98.60:46393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:44607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:08.306597 23121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:08.307542 23121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 18:42:08.315485 23129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:08.315491 23126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:08.315497 23127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:08.315696 23121 server_base.cc:1034] running on GCE node
I20260619 18:42:08.316357 23121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:08.317425 23121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:08.318598 23121 hybrid_clock.cc:648] HybridClock initialized: now 1781894528318579 us; error 31 us; skew 500 ppm
Jun 19 18:42:08 dist-test-slave-rxv6 krb5kdc[23033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:08.321571 23121 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 18:42:08.322522 23121 webserver.cc:460] Webserver started at http://127.0.98.61:41015/ using document root <none> and password file <none>
I20260619 18:42:08.323057 23121 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:08.323117 23121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:08.323320 23121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:08.325387 23121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-1/data/instance:
uuid: "c0cda157412443fc88379592c4cebe66"
format_stamp: "Formatted at 2026-06-19 18:42:08 on dist-test-slave-rxv6"
server_key: "15162b8d46068408d7aa2bff7bbeed95"
server_key_iv: "a4845b681758571ef375c5c55e1daabd"
server_key_version: "encryptionkey@0"
I20260619 18:42:08.325980 23121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-1/wal/instance:
uuid: "c0cda157412443fc88379592c4cebe66"
format_stamp: "Formatted at 2026-06-19 18:42:08 on dist-test-slave-rxv6"
server_key: "15162b8d46068408d7aa2bff7bbeed95"
server_key_iv: "a4845b681758571ef375c5c55e1daabd"
server_key_version: "encryptionkey@0"
I20260619 18:42:08.329639 23121 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 18:42:08.332085 23136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:08.333323 23121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 18:42:08.333426 23121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-1/wal
uuid: "c0cda157412443fc88379592c4cebe66"
format_stamp: "Formatted at 2026-06-19 18:42:08 on dist-test-slave-rxv6"
server_key: "15162b8d46068408d7aa2bff7bbeed95"
server_key_iv: "a4845b681758571ef375c5c55e1daabd"
server_key_version: "encryptionkey@0"
I20260619 18:42:08.333523 23121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:08.345445 23121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:08.348403 23121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:08.348567 23121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:08.355037 23121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:44607
I20260619 18:42:08.355299 23188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:44607 every 8 connection(s)
I20260619 18:42:08.355993 23121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-1/data/info.pb
I20260619 18:42:08.358696 23189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:08.360585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 23121
I20260619 18:42:08.360729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-1/wal/instance
I20260619 18:42:08.361058 392 external_mini_cluster.cc:1428] Setting key 3f3c01a76c2cae22fd8001d55194c7bf
I20260619 18:42:08.363122 23189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 18:42:08 dist-test-slave-rxv6 krb5kdc[23033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:42:08.376783 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.364996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.0.98.62:35977)
0619 18:42:08.365116 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.365120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.366230 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 18:42:08.367176 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.367180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.367246 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.367325 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.368471 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.369007 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.370328 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.370598 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.370710 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.373434 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:08.373437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:08.373439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:08.373467 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:08.374822 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.375304 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.375307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.375308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.375381 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.375685 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.375688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.375690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.375868 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:08.375953 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:08.376474 (+ 521us) server_negotiation.cc:299] Negotiation successful
0619 18:42:08.376601 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 18:42:08.377581 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.365526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35977 (local address 127.0.0.1:54888)
0619 18:42:08.366072 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.366117 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:08.366968 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:08.367498 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:08.367505 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:08.367847 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:08.368326 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.368339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.369129 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.369132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.370209 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.370220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.370778 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.370782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.370953 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.371552 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:08.371573 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:08.373263 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:08.374930 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.374937 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.374953 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.375210 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.375460 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.375462 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.375465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.375598 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.375964 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:08.375971 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:08.376234 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 18:42:08.376404 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 18:42:08.376737 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":149,"threads_started":1}
I20260619 18:42:08.378892 23189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:42:08 dist-test-slave-rxv6 krb5kdc[23033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 18:42:08.386224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
/tmp/dist-test-taskqHyw_j/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46393
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35977,127.0.98.61:44607,127.0.98.60:46393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 18:42:08.387161 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.379363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44607 (local address 127.0.0.1:53596)
0619 18:42:08.379474 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.379487 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:08.379552 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:08.380079 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:08.380081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:08.380241 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:08.380322 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.380327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.381253 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.381256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.381768 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.381775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.382385 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.382389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.382482 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.382890 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:08.382901 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:08.384024 (+ 1123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:08.385669 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.385672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.385673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.385878 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.386195 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.386197 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.386198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.386274 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.386767 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:08.386769 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:08.386849 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 18:42:08.386925 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 18:42:08.387036 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 18:42:08.387322 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.379480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.0.98.61:44607)
0619 18:42:08.379769 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.379778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.379793 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:42:08.379823 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.379827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.379969 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.380081 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.380416 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.381160 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.381863 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.382284 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.382399 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.384150 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:08.384158 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:08.384170 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:08.384189 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:08.385567 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.386004 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.386010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.386012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.386100 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.386359 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.386363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.386364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.386665 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:08.386787 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:08.387014 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 18:42:08.387166 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":122,"threads_started":1}
I20260619 18:42:08.389098 23189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:42:08.389789 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.389555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46393 (local address 127.0.0.1:40630)
0619 18:42:08.389667 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.389732 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 18:42:08.390007 23137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
I20260619 18:42:08.390822 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.390639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46393 (local address 127.0.0.1:40644)
0619 18:42:08.390731 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.390774 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 18:42:08.390937 23189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46393: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
I20260619 18:42:08.392132 23114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } attempt: 3
Jun 19 18:42:08 dist-test-slave-rxv6 krb5kdc[23033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 18:42:08.400053 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.392758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.0.98.61:44607)
0619 18:42:08.393098 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.393101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.393111 (+ 10us) server_negotiation.cc:407] Connection header received
0619 18:42:08.393148 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.393150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.393185 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.393247 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.393662 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.394225 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.394990 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.395228 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.395312 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.397316 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:08.397320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:08.397321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:08.397344 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:08.398613 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.399055 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.399057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.399059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.399121 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.399372 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.399374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.399375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.399530 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:08.399595 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:08.399847 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 18:42:08.399936 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274}
I20260619 18:42:08.400101 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.392643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44607 (local address 127.0.0.1:53598)
0619 18:42:08.392826 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.392840 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:08.392928 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:08.393291 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:08.393294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:08.393457 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:08.393560 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.393565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.394346 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.394350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.394889 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.394898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.395324 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.395327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.395435 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.395945 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:08.395956 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:08.397188 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:08.398718 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.398721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.398724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.398975 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.399209 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.399212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.399214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.399302 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.399598 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:08.399601 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:08.399709 (+ 108us) client_negotiation.cc:769] Sending connection context
0619 18:42:08.399800 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 18:42:08.399934 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 18:42:08.401198 23114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:42:08.401875 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.401631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46393 (local address 127.0.0.1:40654)
0619 18:42:08.401751 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.401806 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 18:42:08.402696 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.402428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46393 (local address 127.0.0.1:40662)
0619 18:42:08.402575 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.402625 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260619 18:42:08.402809 23114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46393: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
I20260619 18:42:08.433966 23114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } attempt: 1
I20260619 18:42:08.434890 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.434539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46393 (local address 127.0.0.1:40670)
0619 18:42:08.434713 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.434793 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 18:42:08.435690 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.435456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46393 (local address 127.0.0.1:40674)
0619 18:42:08.435560 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.435615 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 18:42:08.435823 23114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46393: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
I20260619 18:42:08.440174 23189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } attempt: 1
I20260619 18:42:08.440891 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.440619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46393 (local address 127.0.0.1:40684)
0619 18:42:08.440743 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.440797 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 18:42:08.441515 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.441315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46393 (local address 127.0.0.1:40694)
0619 18:42:08.441411 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.441459 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 18:42:08.441648 23189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46393: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
W20260619 18:42:08.498930 23195 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 18:42:08.499224 23195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 18:42:08.499257 23195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 18:42:08.499316 23195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 18:42:08.503355 23195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 18:42:08.503412 23195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 18:42:08.503437 23195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 18:42:08.503466 23195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 18:42:08.503496 23195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 18:42:08.508816 23195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35977,127.0.98.61:44607,127.0.98.60:46393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 17:59:53 UTC on ccycloud.quasar-efnmxv.root.comops.site
build id 4456541
I20260619 18:42:08.509785 23195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 18:42:08.509786 23189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } attempt: 2
I20260619 18:42:08.510543 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.510282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46393 (local address 127.0.0.1:40708)
0619 18:42:08.510415 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.510471 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 18:42:08.510793 23195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 18:42:08.510969 23114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } attempt: 2
I20260619 18:42:08.511178 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.510981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46393 (local address 127.0.0.1:40716)
0619 18:42:08.511076 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.511121 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 18:42:08.511497 23189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46393: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
I20260619 18:42:08.511608 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.511348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46393 (local address 127.0.0.1:40722)
0619 18:42:08.511480 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.511539 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 18:42:08.512254 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.512050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46393 (local address 127.0.0.1:40732)
0619 18:42:08.512147 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.512196 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 18:42:08.512404 23114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46393: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46393: connect: Connection refused (error 111)
W20260619 18:42:08.518999 23204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:08.519011 23201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 18:42:08.519011 23202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 18:42:08.519353 23195 server_base.cc:1034] running on GCE node
I20260619 18:42:08.519887 23195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 18:42:08.520951 23195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 18:42:08.522107 23195 hybrid_clock.cc:648] HybridClock initialized: now 1781894528522093 us; error 24 us; skew 500 ppm
Jun 19 18:42:08 dist-test-slave-rxv6 krb5kdc[23033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 18:42:08.524691 23195 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 18:42:08.525578 23195 webserver.cc:460] Webserver started at http://127.0.98.60:45205/ using document root <none> and password file <none>
I20260619 18:42:08.526144 23195 fs_manager.cc:359] Metadata directory not provided
I20260619 18:42:08.526211 23195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 18:42:08.526409 23195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 18:42:08.528400 23195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-2/data/instance:
uuid: "6cda02f38f5345f9adb9be8bd06695d7"
format_stamp: "Formatted at 2026-06-19 18:42:08 on dist-test-slave-rxv6"
server_key: "23f4b3f27ad13cb2bdb8289557177a85"
server_key_iv: "010c13a3aae46ab590e886c77d52ccda"
server_key_version: "encryptionkey@0"
I20260619 18:42:08.528931 23195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-2/wal/instance:
uuid: "6cda02f38f5345f9adb9be8bd06695d7"
format_stamp: "Formatted at 2026-06-19 18:42:08 on dist-test-slave-rxv6"
server_key: "23f4b3f27ad13cb2bdb8289557177a85"
server_key_iv: "010c13a3aae46ab590e886c77d52ccda"
server_key_version: "encryptionkey@0"
I20260619 18:42:08.532670 23195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 18:42:08.535203 23211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 18:42:08.536563 23195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 18:42:08.536688 23195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-2/wal
uuid: "6cda02f38f5345f9adb9be8bd06695d7"
format_stamp: "Formatted at 2026-06-19 18:42:08 on dist-test-slave-rxv6"
server_key: "23f4b3f27ad13cb2bdb8289557177a85"
server_key_iv: "010c13a3aae46ab590e886c77d52ccda"
server_key_version: "encryptionkey@0"
I20260619 18:42:08.536798 23195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 18:42:08.548616 23195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 18:42:08.551651 23195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 18:42:08.551824 23195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 18:42:08.558568 23195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46393
I20260619 18:42:08.558828 23263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46393 every 8 connection(s)
I20260619 18:42:08.559506 23195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-2/data/info.pb
I20260619 18:42:08.562201 23264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 18:42:08.563266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu as pid 23195
I20260619 18:42:08.563411 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqHyw_j/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781894249862417-392-0/minicluster-data/master-2/wal/instance
I20260619 18:42:08.563750 392 external_mini_cluster.cc:1428] Setting key 09de99d850fb1698979202bf7d3d50af
I20260619 18:42:08.566757 23264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:42:08 dist-test-slave-rxv6 krb5kdc[23033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 18:42:08 dist-test-slave-rxv6 krb5kdc[23033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:42:08.577142 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.568653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54892 (local address 127.0.98.62:35977)
0619 18:42:08.568793 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.568798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.569148 (+ 350us) server_negotiation.cc:407] Connection header received
0619 18:42:08.569215 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.569218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.569272 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.569353 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.570030 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.570499 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.571577 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.571915 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.572021 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.574228 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:08.574230 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:08.574232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:08.574257 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:08.575585 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.575986 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.575989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.575991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.576068 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.576303 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.576306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.576307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.576512 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:08.576613 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:08.576918 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 18:42:08.577012 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 18:42:08.577629 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.568565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35977 (local address 127.0.0.1:54892)
0619 18:42:08.568947 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.568995 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:08.569126 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:08.569413 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:08.569419 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:08.569764 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:08.569878 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.569883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.570608 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.570611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.571455 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.571465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.572050 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.572053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.572139 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.572581 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:08.572600 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:08.574083 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:08.575676 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.575684 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.575690 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.575898 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.576147 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.576150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.576152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.576226 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.576623 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:08.576629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:08.576739 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 18:42:08.576876 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 18:42:08.577051 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":36,"thread_start_us":117,"threads_started":1}
I20260619 18:42:08.577629 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.565041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40746 (local address 127.0.98.60:46393)
0619 18:42:08.565463 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.565483 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.565529 (+ 46us) server_negotiation.cc:407] Connection header received
0619 18:42:08.566365 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.566389 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.566825 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.567184 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.567843 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.569120 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.570020 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.570484 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.570653 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.573086 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:08.573093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:08.573106 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:08.573132 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:08.574648 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.575069 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.575075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.575081 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.575187 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.575483 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.575485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.575486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.575861 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:08.576037 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:08.576329 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 18:42:08.576634 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":254,"thread_start_us":105,"threads_started":1}
I20260619 18:42:08.578902 23264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 18:42:08 dist-test-slave-rxv6 krb5kdc[23033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 18:42:08 dist-test-slave-rxv6 krb5kdc[23033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 18:42:08.587425 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.579383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:44607 (local address 127.0.0.1:53610)
0619 18:42:08.579643 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.579657 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:08.579739 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:08.579989 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:08.579992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:08.580193 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:08.580312 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.580318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.581263 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.581266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.582092 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.582101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.582601 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.582603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.582671 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.583139 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:08.583153 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:08.584323 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:08.585955 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.585959 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.585961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.586205 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.586490 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.586494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.586496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.586588 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.586928 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:08.586932 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:08.587030 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 18:42:08.587150 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 18:42:08.587275 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260619 18:42:08.587706 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.579509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.0.98.61:44607)
0619 18:42:08.579623 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.579626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.579698 (+ 72us) server_negotiation.cc:407] Connection header received
0619 18:42:08.579825 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.579828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.579877 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.579964 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.580410 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.581154 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.582184 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.582509 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.582607 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.584448 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:08.584452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:08.584455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:08.584483 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:08.585805 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.586302 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.586305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.586306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.586398 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.586669 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.586671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.586673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.586841 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:08.586914 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:08.587240 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 18:42:08.587369 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 18:42:08.588343 23264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } has no permanent_uuid. Determining permanent_uuid...
I20260619 18:42:08.589035 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.580211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54900 (local address 127.0.98.62:35977)
0619 18:42:08.580313 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.580317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.580329 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:42:08.580446 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.580448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.580486 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.580548 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.581205 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.582012 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.582971 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.583255 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.583355 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.585639 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:08.585642 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:08.585644 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:08.585667 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:08.587195 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.587656 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.587660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.587662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.587748 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.588034 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.588037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.588039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.588263 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:08.588480 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:08.588789 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 18:42:08.588895 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Jun 19 18:42:08 dist-test-slave-rxv6 krb5kdc[23033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 18:42:08 dist-test-slave-rxv6 krb5kdc[23033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 18:42:08.595896 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.588789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46393 (local address 127.0.0.1:40760)
0619 18:42:08.588921 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.588932 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:08.589003 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:08.589217 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:08.589220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:08.589359 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:08.589421 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.589425 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.590248 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.590252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.590738 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.590745 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.591163 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.591166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.591231 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.591647 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:08.591657 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:08.592774 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:08.594527 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.594531 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.594533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.594766 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.594998 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.595001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.595003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.595091 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.595411 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:08.595414 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:08.595489 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 18:42:08.595597 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 18:42:08.595723 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 18:42:08.595963 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.588825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.0.98.60:46393)
0619 18:42:08.588916 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.588921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.589080 (+ 159us) server_negotiation.cc:407] Connection header received
0619 18:42:08.589105 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.589107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.589147 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.589204 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.589512 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.590164 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.590830 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.591077 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.591157 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.592886 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:08.592889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:08.592892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:08.592916 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:08.594414 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.594854 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.594856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.594858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.594917 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.595172 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.595174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.595175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.595327 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:08.595407 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:08.595649 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 18:42:08.595774 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":33}
I20260619 18:42:08.598655 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.590122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53620 (local address 127.0.98.61:44607)
0619 18:42:08.590218 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.590222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.590234 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:42:08.590333 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.590336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.590376 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.590439 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.590991 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.591652 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.592386 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.592700 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.592801 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.595438 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:08.595441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:08.595444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:08.595464 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:08.597039 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.597518 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.597521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.597523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.597607 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.597861 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.597864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.597865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.598117 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:08.598336 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:08.598401 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 18:42:08.598503 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 18:42:08.599917 23264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7: Bootstrap starting.
I20260619 18:42:08.601807 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 18:42:08.602391 23264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:08.602650 23189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } attempt: 3
I20260619 18:42:08.603251 23264 log.cc:824] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:08.603123 23088 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d9f1582d9d0140339c83d5ecafe1d555"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54900
I20260619 18:42:08.605291 23264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7: No bootstrap required, opened a new log
Jun 19 18:42:08 dist-test-slave-rxv6 krb5kdc[23033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 18:42:08.608875 23264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } }
I20260619 18:42:08.609138 23264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cda02f38f5345f9adb9be8bd06695d7, State: Initialized, Role: FOLLOWER
I20260619 18:42:08.609485 23264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } }
I20260619 18:42:08.610059 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } } }
I20260619 18:42:08.610201 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:08.610982 23264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:08.611550 23114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } attempt: 3
I20260619 18:42:08.612989 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.603000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46393 (local address 127.0.0.1:40776)
0619 18:42:08.604508 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.604522 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:08.604601 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:08.604937 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:08.604940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:08.605158 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:08.605261 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.605267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.606216 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.606219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.607047 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.607055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.607591 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.607594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.607676 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.608146 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:08.608159 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:08.609324 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:08.611254 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.611258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.611260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.611506 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.611881 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.611884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.611886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.611976 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.612518 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:08.612521 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:08.612619 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 18:42:08.612696 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 18:42:08.612829 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1409}
I20260619 18:42:08.613699 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.603058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.0.98.60:46393)
0619 18:42:08.604737 (+ 1679us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.604741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.604753 (+ 12us) server_negotiation.cc:407] Connection header received
0619 18:42:08.604783 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.604786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.604823 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.604894 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.605386 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.606117 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.607167 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.607489 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.607593 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.609448 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:08.609452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:08.609455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:08.609479 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:08.611147 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.611628 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.611632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.611634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.611778 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.612106 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.612138 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.612140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.612404 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:08.613318 (+ 914us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:08.613392 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 18:42:08.613504 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1612}
W20260619 18:42:08.616477 23282 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:42:08.616561 23282 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:42:08.616823 23189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66: Bootstrap starting.
I20260619 18:42:08.614925 23195 master_runner.cc:186] Error getting master registration for 127.0.98.62:35977: OK, instance_id { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" instance_seqno: 1781894528150165 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:42:08.618178 23195 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:42:08.619617 23189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66: Neither blocks nor log segments found. Creating new log.
I20260619 18:42:08.620487 23189 log.cc:824] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66: Log is configured to *not* fsync() on all Append() calls
Jun 19 18:42:08 dist-test-slave-rxv6 krb5kdc[23033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781894528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 18:42:08.622687 23189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66: No bootstrap required, opened a new log
I20260619 18:42:08.626984 23189 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } }
I20260619 18:42:08.627233 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.612014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40778 (local address 127.0.98.60:46393)
0619 18:42:08.612624 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.612635 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.612648 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:42:08.612680 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.612683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.612725 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.612848 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.613377 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.618790 (+ 5413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.620033 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.620409 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.620545 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.622964 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:08.622968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:08.622972 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:08.623001 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:08.625314 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.625826 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.625830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.625832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.625981 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.626284 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.626287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.626288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.626506 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:08.626600 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:08.626912 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 18:42:08.627040 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":99,"threads_started":1}
I20260619 18:42:08.627218 23189 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0cda157412443fc88379592c4cebe66, State: Initialized, Role: FOLLOWER
I20260619 18:42:08.627164 23118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.611981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46393 (local address 127.0.0.1:40778)
0619 18:42:08.612193 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 18:42:08.612207 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 18:42:08.612283 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 18:42:08.612893 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 18:42:08.612908 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 18:42:08.613082 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 18:42:08.613196 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.613205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.619003 (+ 5798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.619031 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.619892 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 18:42:08.619908 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.620581 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.620586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 18:42:08.620682 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.621228 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 18:42:08.621246 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 18:42:08.622804 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 18:42:08.625458 (+ 2654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.625463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.625465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.625716 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.626083 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 18:42:08.626087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 18:42:08.626089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 18:42:08.626187 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 18:42:08.626611 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 18:42:08.626614 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 18:42:08.626730 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 18:42:08.626837 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 18:42:08.626981 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260619 18:42:08.627552 23189 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } }
I20260619 18:42:08.628183 23285 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } } }
I20260619 18:42:08.628350 23285 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:08.630478 23114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555: Bootstrap starting.
I20260619 18:42:08.630884 23189 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:08.633602 23114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555: Neither blocks nor log segments found. Creating new log.
W20260619 18:42:08.637390 23296 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:42:08.637581 23296 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:42:08.634276 23121 master_runner.cc:186] Error getting master registration for 127.0.98.62:35977: OK, instance_id { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" instance_seqno: 1781894528150165 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 18:42:08.637900 23121 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 18:42:08.638634 23114 log.cc:824] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555: Log is configured to *not* fsync() on all Append() calls
I20260619 18:42:08.641036 23114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555: No bootstrap required, opened a new log
I20260619 18:42:08.644282 23114 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } }
I20260619 18:42:08.644526 23114 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f1582d9d0140339c83d5ecafe1d555, State: Initialized, Role: FOLLOWER
I20260619 18:42:08.644865 23114 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } }
I20260619 18:42:08.645469 23299 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } } }
I20260619 18:42:08.645732 23299 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:08.646319 23114 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 18:42:08.646401 23088 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 18:42:08.646520 23088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [term 0 FOLLOWER]: Advancing to term 1
W20260619 18:42:08.646742 23067 tablet.cc:2369] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 18:42:08.647742 23088 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } }
I20260619 18:42:08.648670 23088 leader_election.cc:288] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [CANDIDATE]: Term 1 election: Requested vote from peers c0cda157412443fc88379592c4cebe66 (127.0.98.61:44607), 6cda02f38f5345f9adb9be8bd06695d7 (127.0.98.60:46393)
I20260619 18:42:08.649715 23163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9f1582d9d0140339c83d5ecafe1d555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c0cda157412443fc88379592c4cebe66"
I20260619 18:42:08.649984 23163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:08.651275 23163 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9f1582d9d0140339c83d5ecafe1d555 in term 1.
I20260619 18:42:08.651757 23238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9f1582d9d0140339c83d5ecafe1d555" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6cda02f38f5345f9adb9be8bd06695d7"
I20260619 18:42:08.651998 23238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 18:42:08.653221 23238 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9f1582d9d0140339c83d5ecafe1d555 in term 1.
I20260619 18:42:08.655053 23062 leader_election.cc:302] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cda02f38f5345f9adb9be8bd06695d7, d9f1582d9d0140339c83d5ecafe1d555; no voters:
I20260619 18:42:08.655387 23299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 18:42:08.655570 23299 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [term 1 LEADER]: Becoming Leader. State: Replica: d9f1582d9d0140339c83d5ecafe1d555, State: Running, Role: LEADER
I20260619 18:42:08.655915 23299 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } }
I20260619 18:42:08.657701 23299 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9f1582d9d0140339c83d5ecafe1d555. Latest consensus state: current_term: 1 leader_uuid: "d9f1582d9d0140339c83d5ecafe1d555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } } }
I20260619 18:42:08.657809 23299 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:08.662294 23046 master_runner.cc:263] No leader master found from master d9f1582d9d0140339c83d5ecafe1d555
I20260619 18:42:08.662381 23046 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 18:42:08.665546 23310 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 18:42:08.665632 23310 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 18:42:08.666106 23320 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 18:42:08.666949 23320 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 18:42:08.677858 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.652648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54916 (local address 127.0.98.62:35977)
0619 18:42:08.652762 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.652766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.654407 (+ 1641us) server_negotiation.cc:407] Connection header received
0619 18:42:08.654582 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.654585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.654631 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.654678 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.657566 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.658376 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.664384 (+ 6008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.664732 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.664805 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.671308 (+ 6503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:08.671312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:08.671315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:08.671344 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:08.672896 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.675498 (+ 2602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.675502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.675504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.675593 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.676259 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.676262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.676264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.676501 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:08.676609 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:08.677557 (+ 948us) server_negotiation.cc:299] Negotiation successful
0619 18:42:08.677691 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 18:42:08.681676 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.653327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53632 (local address 127.0.98.61:44607)
0619 18:42:08.653455 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.653459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.655423 (+ 1964us) server_negotiation.cc:407] Connection header received
0619 18:42:08.655817 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.655821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.655866 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.655912 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.660716 (+ 4804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.661502 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.665873 (+ 4371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.666250 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.666324 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.673415 (+ 7091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:08.673420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:08.673422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:08.673454 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:08.675032 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.675876 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.675880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.675882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.675961 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.676898 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.676901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.676903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.677128 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:08.677232 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:08.681383 (+ 4151us) server_negotiation.cc:299] Negotiation successful
0619 18:42:08.681511 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 18:42:08.684199 23238 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7 [term 1 FOLLOWER]: Refusing update from remote peer d9f1582d9d0140339c83d5ecafe1d555: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:08.684927 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:08.654836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40794 (local address 127.0.98.60:46393)
0619 18:42:08.654952 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.654957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.657490 (+ 2533us) server_negotiation.cc:407] Connection header received
0619 18:42:08.659725 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.659728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.659778 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.659824 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.662508 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.663299 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.669198 (+ 5899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:08.669590 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:08.669665 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:08.678433 (+ 8768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 18:42:08.678437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 18:42:08.678440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 18:42:08.678474 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 18:42:08.680083 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.683449 (+ 3366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.683454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.683456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.683560 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 18:42:08.683847 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 18:42:08.683851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 18:42:08.683852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 18:42:08.684087 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 18:42:08.684227 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:08.684649 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 18:42:08.684772 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 18:42:08.684918 23299 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:08.689675 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9f1582d9d0140339c83d5ecafe1d555. Latest consensus state: current_term: 1 leader_uuid: "d9f1582d9d0140339c83d5ecafe1d555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } } }
I20260619 18:42:08.689810 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:08.691483 23163 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66 [term 1 FOLLOWER]: Refusing update from remote peer d9f1582d9d0140339c83d5ecafe1d555: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 18:42:08.692104 23299 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 18:42:08.697463 23285 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9f1582d9d0140339c83d5ecafe1d555. Latest consensus state: current_term: 1 leader_uuid: "d9f1582d9d0140339c83d5ecafe1d555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } } }
I20260619 18:42:08.697604 23285 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:08.699663 23299 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9f1582d9d0140339c83d5ecafe1d555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } } }
I20260619 18:42:08.699795 23299 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:08.700908 23270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9f1582d9d0140339c83d5ecafe1d555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } } }
I20260619 18:42:08.701030 23270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:08.701439 23323 mvcc.cc:205] Tried to move back new op lower bound from 7298639989455851520 to 7298639989378027520. Current Snapshot: MvccSnapshot[applied={T|T < 7298639989455851520}]
I20260619 18:42:08.701604 23322 mvcc.cc:205] Tried to move back new op lower bound from 7298639989455851520 to 7298639989378027520. Current Snapshot: MvccSnapshot[applied={T|T < 7298639989455851520}]
I20260619 18:42:08.705541 23299 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9f1582d9d0140339c83d5ecafe1d555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } } }
I20260619 18:42:08.705675 23299 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555 [sys.catalog]: This master's current role is: LEADER
I20260619 18:42:08.706631 23285 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9f1582d9d0140339c83d5ecafe1d555" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9f1582d9d0140339c83d5ecafe1d555" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } peers { permanent_uuid: "c0cda157412443fc88379592c4cebe66" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 44607 } } peers { permanent_uuid: "6cda02f38f5345f9adb9be8bd06695d7" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46393 } } }
I20260619 18:42:08.706748 23285 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 18:42:08.706915 23325 mvcc.cc:205] Tried to move back new op lower bound from 7298639989455851520 to 7298639989378027520. Current Snapshot: MvccSnapshot[applied={T|T < 7298639989455851520}]
I20260619 18:42:08.710616 23320 catalog_manager.cc:1347] Generated new cluster ID: eae3173060f54242a0c19f109a4759d9
I20260619 18:42:08.710681 23320 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 18:42:08.736892 23320 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 18:42:08.738443 23320 catalog_manager.cc:1504] Loading token signing keys...
I20260619 18:42:08.764168 23320 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9f1582d9d0140339c83d5ecafe1d555: Generated new TSK 0
I20260619 18:42:08.765055 23320 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 18:42:08.796924 23117 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:08.793550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54922 (local address 127.0.98.62:35977)
0619 18:42:08.793692 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.793697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.795722 (+ 2025us) server_negotiation.cc:407] Connection header received
0619 18:42:08.795952 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.795956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.796015 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.796076 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.796769 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54922: BlockingRecv error: recv got EOF from 127.0.0.1:54922 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 18:42:08.798638 23194 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:08.794381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.0.98.61:44607)
0619 18:42:08.794505 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.794510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.797433 (+ 2923us) server_negotiation.cc:407] Connection header received
0619 18:42:08.797608 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.797611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.797663 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.797717 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.798513 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53640: BlockingRecv error: recv got EOF from 127.0.0.1:53640 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 18:42:08.799970 23276 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:08.795084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.0.98.60:46393)
0619 18:42:08.795196 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.795200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.798133 (+ 2933us) server_negotiation.cc:407] Connection header received
0619 18:42:08.799002 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.799005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.799059 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.799114 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.799843 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40806: BlockingRecv error: recv got EOF from 127.0.0.1:40806 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 18:42:08.804854 23194 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:08.803238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53650 (local address 127.0.98.61:44607)
0619 18:42:08.803361 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.803366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.803874 (+ 508us) server_negotiation.cc:407] Connection header received
0619 18:42:08.804085 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.804088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.804160 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.804211 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.804744 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53650: BlockingRecv error: recv got EOF from 127.0.0.1:53650 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 18:42:08.809846 23194 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:08.808422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53652 (local address 127.0.98.61:44607)
0619 18:42:08.808521 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.808525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.808997 (+ 472us) server_negotiation.cc:407] Connection header received
0619 18:42:08.809168 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.809171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.809220 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.809266 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.809741 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53652: BlockingRecv error: recv got EOF from 127.0.0.1:53652 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 18:42:08.819150 23194 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:08.814751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.0.98.61:44607)
0619 18:42:08.814856 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.814860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.818246 (+ 3386us) server_negotiation.cc:407] Connection header received
0619 18:42:08.818383 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.818387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.818442 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.818530 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.819026 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53656: BlockingRecv error: recv got EOF from 127.0.0.1:53656 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 18:42:08.826601 23194 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:08.825354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53658 (local address 127.0.98.61:44607)
0619 18:42:08.825517 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.825522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.825670 (+ 148us) server_negotiation.cc:407] Connection header received
0619 18:42:08.825829 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.825833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.825891 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.825996 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.826480 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53658: BlockingRecv error: recv got EOF from 127.0.0.1:53658 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260619 18:42:08.837642 23194 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:08.836497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.0.98.61:44607)
0619 18:42:08.836612 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.836618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.836787 (+ 169us) server_negotiation.cc:407] Connection header received
0619 18:42:08.836931 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.836934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.837004 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.837076 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.837525 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53672: BlockingRecv error: recv got EOF from 127.0.0.1:53672 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260619 18:42:08.856900 23194 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:08.855847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.0.98.61:44607)
0619 18:42:08.856003 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.856009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.856082 (+ 73us) server_negotiation.cc:407] Connection header received
0619 18:42:08.856203 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.856207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.856269 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.856378 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.856779 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53680: BlockingRecv error: recv got EOF from 127.0.0.1:53680 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 18:42:08.891889 23194 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:08.890852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53688 (local address 127.0.98.61:44607)
0619 18:42:08.891003 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.891009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.891148 (+ 139us) server_negotiation.cc:407] Connection header received
0619 18:42:08.891261 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.891263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.891311 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.891380 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.891763 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53688: BlockingRecv error: recv got EOF from 127.0.0.1:53688 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 18:42:08.958978 23194 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:08.957845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.0.98.61:44607)
0619 18:42:08.958023 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:08.958028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:08.958122 (+ 94us) server_negotiation.cc:407] Connection header received
0619 18:42:08.958281 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:08.958284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:08.958332 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:08.958401 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:08.958862 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53690: BlockingRecv error: recv got EOF from 127.0.0.1:53690 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 18:42:09.090225 23194 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:09.089270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53696 (local address 127.0.98.61:44607)
0619 18:42:09.089398 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:09.089402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:09.089490 (+ 88us) server_negotiation.cc:407] Connection header received
0619 18:42:09.089612 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:09.089614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:09.089663 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:09.089743 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:09.090125 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53696: BlockingRecv error: recv got EOF from 127.0.0.1:53696 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 18:42:09.349871 23194 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 18:42:09.348727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53704 (local address 127.0.98.61:44607)
0619 18:42:09.348898 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:09.348902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:09.348963 (+ 61us) server_negotiation.cc:407] Connection header received
0619 18:42:09.349103 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:09.349106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:09.349160 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:09.349231 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 18:42:09.349709 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53704: BlockingRecv error: recv got EOF from 127.0.0.1:53704 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 18:42:09.618997 23282 catalog_manager.cc:1259] Loaded cluster ID: eae3173060f54242a0c19f109a4759d9
I20260619 18:42:09.619086 23282 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7: loading cluster ID for follower catalog manager: success
I20260619 18:42:09.621027 23282 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7: acquiring CA information for follower catalog manager: success
I20260619 18:42:09.622413 23282 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6cda02f38f5345f9adb9be8bd06695d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:42:09.639168 23296 catalog_manager.cc:1259] Loaded cluster ID: eae3173060f54242a0c19f109a4759d9
I20260619 18:42:09.639257 23296 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66: loading cluster ID for follower catalog manager: success
I20260619 18:42:09.641113 23296 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66: acquiring CA information for follower catalog manager: success
I20260619 18:42:09.642498 23296 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c0cda157412443fc88379592c4cebe66: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 18:42:09.868482 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:09.864275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53712 (local address 127.0.98.61:44607)
0619 18:42:09.864580 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:09.864586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:09.864604 (+ 18us) server_negotiation.cc:407] Connection header received
0619 18:42:09.864773 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:09.864776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:09.864935 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:09.865046 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:09.865327 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.865900 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.866844 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.867095 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.867184 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:09.867508 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:09.868060 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:09.868189 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:09.868247 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 18:42:09.868309 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":87,"threads_started":1}
I20260619 18:42:09.874985 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:09.871496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40808 (local address 127.0.98.60:46393)
0619 18:42:09.871729 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:09.871733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:09.871747 (+ 14us) server_negotiation.cc:407] Connection header received
0619 18:42:09.871837 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:09.871839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:09.871965 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:09.872063 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:09.872213 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.872822 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.873617 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.873850 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.873967 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:09.874186 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:09.874681 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:09.874768 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:09.874794 (+ 26us) server_negotiation.cc:299] Negotiation successful
0619 18:42:09.874841 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
I20260619 18:42:09.881724 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:09.877790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40822 (local address 127.0.98.60:46393)
0619 18:42:09.877942 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:09.877946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:09.878252 (+ 306us) server_negotiation.cc:407] Connection header received
0619 18:42:09.878418 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:09.878421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:09.878471 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:09.878516 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:09.878944 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.879436 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.880494 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.880739 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.880826 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:09.881310 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:09.881386 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:09.881451 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:09.881531 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 18:42:09.881588 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 18:42:09.882179 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:09.877750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54930 (local address 127.0.98.62:35977)
0619 18:42:09.878047 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:09.878051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:09.878066 (+ 15us) server_negotiation.cc:407] Connection header received
0619 18:42:09.878283 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:09.878286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:09.878465 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:09.878587 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:09.878812 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.879678 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.880677 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.880969 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.881054 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:09.881344 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:09.881681 (+ 337us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:09.881767 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:09.881923 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 18:42:09.881994 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":73,"threads_started":1}
I20260619 18:42:09.882927 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 18:42:09.878271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.0.98.61:44607)
0619 18:42:09.878391 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 18:42:09.878395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 18:42:09.878408 (+ 13us) server_negotiation.cc:407] Connection header received
0619 18:42:09.878498 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 18:42:09.878501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 18:42:09.878550 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 18:42:09.878614 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 18:42:09.878914 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.879678 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.881415 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 18:42:09.881733 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 18:42:09.881831 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 18:42:09.882451 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 18:42:09.882535 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 18:42:09.882590 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 18:42:09.882737 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 18:42:09.882796 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 18:42:09.885515 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 23046
I20260619 18:42:09.892256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 23121
I20260619 18:42:09.898167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqHyw_j/build/debug/bin/kudu with pid 23195
2026-06-19T18:42:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2985 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5976 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (280026 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST